> starting XS agent manually Either unmanage/manage cluster or perform a force reconnect to the host.
Regards, Somesh -----Original Message----- From: tony_caot...@163.com [mailto:tony_caot...@163.com] Sent: Wednesday, August 12, 2015 5:14 AM To: users@cloudstack.apache.org Subject: Re: XenServer is disconnected after CS hosts shutdown After I disabled and enabled XS cluster & primary. it works. seems XS agent have started by ACS host. so what is the correct behavior of starting XS agent manually ? ----------- Cao Tong On 08/12/2015 04:41 PM, tony_caot...@163.com wrote: > > Hello, > > Almost one month gone, my problem is still here. I really really need > someone to help me. > > new Settings ACS 4.4.4 XS 6.2 > After reboot, errors here: > > 2015-08-12 16:18:54,557 INFO [c.c.a.t.Request] > (StatsCollector-3:ctx-b287085a) not building log message for '[{}]', > _cmds.length == 1 > 2015-08-12 16:18:54,557 DEBUG [c.c.a.m.ClusteredAgentAttache] > (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Forwarding > null to 191386435611186 > 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache] > (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Routing from > 249082151178140 > 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentAttache] > (AgentManager-Handler-12:null) Seq 9-4450963806725603425: Link is closed > 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-12:null) Seq 9-4450963806725603425: MgmtId > 249082151178140: Req: Resource [Host:9] is unreachable: Host 9: Link > is closed > 2015-08-12 16:18:54,558 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-12:null) Seq 9--1: MgmtId 249082151178140: Req: > Routing to peer > 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (AgentManager-Handler-15:null) Seq 9--1: MgmtId 249082151178140: Req: > Cancel request received > 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] > (AgentManager-Handler-15:null) Seq 9-4450963806725603425: Cancelling. > 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting > some more time because this is the current command > 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Waiting > some more time because this is the current command > 2015-08-12 16:18:54,559 INFO [c.c.u.e.CSExceptionErrorCode] > (StatsCollector-3:ctx-b287085a) Could not find exception: > com.cloud.exception.OperationTimedoutException in error code list for > exceptions > 2015-08-12 16:18:54,559 INFO [c.c.a.t.Request] > (StatsCollector-3:ctx-b287085a) not building log message for '[{}]', > _cmds.length == 1 > 2015-08-12 16:18:54,559 WARN [c.c.a.m.AgentAttache] > (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Timed out > on null > 2015-08-12 16:18:54,559 DEBUG [c.c.a.m.AgentAttache] > (StatsCollector-3:ctx-b287085a) Seq 9-4450963806725603425: Cancelling. > 2015-08-12 16:18:54,559 DEBUG [c.c.s.StorageManagerImpl] > (StatsCollector-3:ctx-b287085a) Unable to send storage pool command to > Pool[3|NetworkFilesystem] via 9 > com.cloud.exception.OperationTimedoutException: Commands > 4450963806725603425 to Host 9 timed out after 3600 > at > com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:434) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:418) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:965) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:390) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:404) > at > com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > > > ----------- > Cao Tong > > On 07/30/2015 10:55 AM, tony_caot...@163.com wrote: >> >> Hi Yiping, Thanks for your reply first. >> >> >> My NFS server deployed in ACS manager host, another host is a single >> xenserver. >> A KVM environment deployed in ACS manager host too. >> there is three storages named, Kprimary, Xprimary, Secondary. >> >> First, I add KVM cluster with zone-wide Kprimary, it works fine even >> if after reboot. >> Second, I add Xenserver with Xprimary(cluster-wide), it still works >> fine. >> Then, I set zone to disable, shutdown all system VMs, disable >> those two hosts. finally shutdown xenserver host. >> When xenserver's shutdown finish, I stop those services by order ( >> cloudstack-agent, cloudstack-management, libvirtd, nfs, rpcbind, >> mysqld). >> At last shutdown ACS host. >> >> The start process is totally reverse of this order. >> >> when done of starting, ACS says unable to send cmd to pool via host >> xenserver. >> >> I am sure nothing have change in my environment during reboot time. >> When I fix this problem, the only change is change Kprimary from >> zone-wide to cluster-wide. >> >> I guess that after reboot some status have been initial from >> beginning, ACS found that Xenserver host have two primary can be used. >> so it use the high priority one， and it is Kprimary. >> >> Whatever, Maybe it could help you peoples to get ACS better. >> >> BTW, Error logs attached some days before is already point out this >> is a storage problem >> like: >> >> Unable to send storage pool command >> to Pool[4|NetworkFilesystem] via 4 >> >> >> >> ----------- >> Cao Tong >> >> On 07/30/2015 12:44 AM, Yiping Zhang wrote: >>> Well, sometimes people can’t answer a question because of lack of >>> relevant information, or simply because no one has encountered a >>> similar >>> situation before. >>> >>> Looking at your past messages on this thread, there were no mentions >>> about >>> primary storage. Obviously, your primary storage configuration had >>> changed >>> between the time you shut down CS manager and xenservers and the >>> time you >>> restarted them. That is the vital info the list didn’t know. >>> >>> To best of my knowledge, zone wide primary storage has never been >>> supported for Xen hypervisors. >>> >>> I do have to say that quite often CloudStack error messages are very >>> cryptic, do not provide enough *useful* information to help users >>> identify >>> and trouble shoot actual problems. Those stack trace output might be a >>> gold mine to developers, but they are utterly useless for end users. >>> >>> Just my $0.02 >>> >>> Yiping >>> >>> On 7/28/15, 11:19 PM, "tony_caot...@163.com" <tony_caot...@163.com> >>> wrote: >>> >>>> Hi, Finally I resolved this problem by my self. >>>> >>>> * Primary Storage: A storage resource typically provided to a single >>>> cluster for the actual running of instance disk images. (Zone-wide >>>> primary storage is an option, though not typically used.) >>>> >>>> This line above is from >>>> http://docs.cloudstack.apache.org/en/master/concepts.html >>>> >>>> Because I have a Zone-wide primary storage, ACS can not find the >>>> correct >>>> primary which belong to XenServer cluster after reboot. >>>> >>>> Then I change the Zone-wide primary to cluster-wide, it resolved. >>>> >>>> Right now, I have two primary storage, one is kvm cluster-wide, >>>> another >>>> is xenserver cluster-wide. >>>> >>>> Above is for people who have the same problem oneday. >>>> >>>> by the way, I am very curious why I never receive replys from this >>>> a big >>>> community ?? of course except the very beginning. >>>> >>>> Is my English skill really really poor, result in no body can >>>> understood >>>> what language I am speaking ? >>>> >>>> ----------- >>>> Cao Tong >>>> >>>> On 07/22/2015 09:03 PM, tony_caot...@163.com wrote: >>>>> Hey! help please... >>>>> >>>>> some news. >>>>> I think the cause is that the ACS host can't communicate with >>>>> XenServer host. >>>>> ACS continues outputing logs like this >>>>> >>>>> 2015-07-22 20:42:13,555 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>> (AgentManager-Handler-7:null) Seq 5-8174877748607582212: Forwarding >>>>> Seq 5-8174877748607582212: { Cmd , MgmtId: 279278805451459, via: 5, >>>>> Ver: v1, Flags: 100111, >>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>> 280345368052992 >>>>> >>>>> I am not sure that if the ACS status is wrong or some services on >>>>> xenserver are not opend. >>>>> >>>>> on xenserver , I found *xenheartbeat.sh is not running.* >>>>> *(/bin/bash /opt/cloud/bin/xenheartbeat.sh >>>>> 00d8e0d0-8561-4b3d-9044-cbc496ff22cc 120 60)* >>>>> >>>>> As some operations about xenserver was pending， xenserver can not be >>>>> deleted from web UI. >>>>> >>>>> I got a temporary solution >>>>> >>>>> 1. delete jobs from DB cloud.vm_work_job. >>>>> 2. delete xenserver from DB cloud.host. >>>>> 3. add xenserver host back from web UI. >>>>> >>>>> then it works. >>>>> >>>>> Does anyone have a idea for this? >>>>> >>>>> Could anyone tell what things does ACS do on xenserver host when >>>>> adding a xenserver ? >>>>> >>>>> Thanks, >>>>> >>>>> ----------- >>>>> Cao Tong >>>>> >>>>> On 07/22/2015 04:26 PM, tony_caot...@163.com wrote: >>>>>> @prashant, following it the answer of you questions >>>>>> >>>>>> 1. Yes, primary storage is connected fine for my xenserver. >>>>>> >>>>>> 2. No, Xenserver's password is not changed. >>>>>> >>>>>> 3. yes, web UI is fine, and I can login. >>>>>> >>>>>> 4. before reboot, I unmanaged and disabled resources, and after >>>>>> reboot I have enabled all of them. >>>>>> >>>>>> 5. hosts is states is UP. >>>>>> >>>>>> 6. No yum update in anywhere. >>>>>> >>>>>> 7. system VMs status is fine, i think. >>>>>> >>>>>> ----------- >>>>>> Cao Tong >>>>>> >>>>>> On 07/22/2015 04:13 PM, tony_caot...@163.com wrote: >>>>>>> Hi, >>>>>>> >>>>>>> After reinstall, I got the problem again >>>>>>> >>>>>>> So, I will describe once again. >>>>>>> >>>>>>> WHAT my environment looks like: >>>>>>> >>>>>>> I have a ACS server host and a xenserver host, After both reboot, I >>>>>>> can not create a VM on xenserver through ACS. >>>>>>> A KVM and A NFS are running together in ACS manager host. >>>>>>> >>>>>>> the status of new VM is always 'staring' on the WEB, but I can >>>>>>> create new VM using xencenter. >>>>>>> >>>>>>> ------------- ERR LOGS ---------- >>>>>>> 2015-07-22 15:56:56,357 DEBUG [c.c.s.StorageManagerImpl] >>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to send storage pool command >>>>>>> to Pool[4|NetworkFilesystem] via 4 >>>>>>> com.cloud.exception.OperationTimedoutException: Commands >>>>>>> 2829104990918803478 to Host 4 timed out after 3600 >>>>>>> >>>>>>> 2015-07-22 15:56:56,358 INFO [c.c.s.StatsCollector] >>>>>>> (StatsCollector-3:ctx-1aa2e8c9) Unable to reach >>>>>>> Pool[4|NetworkFilesystem] >>>>>>> com.cloud.exception.StorageUnavailableException: Resource >>>>>>> [StoragePool:4] is unreachable: Unable to send command to the pool >>>>>>> >>>>>>> >>>>>>> ------------- and there are lots of DEBUG infos ------- repeat >>>>>>> again and again ----------- >>>>>>> >>>>>>> 2015-07-22 15:36:12,887 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>> (AgentManager-Handler-14:null) Seq 4-8064821032713715922: >>>>>>> Forwarding >>>>>>> Seq 4-8064821032713715922: { Cmd , MgmtId: 227448510156211, >>>>>>> via: 4, >>>>>>> Ver: v1, Flags: 100111, >>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>>>> 116784073679673 >>>>>>> 2015-07-22 15:36:12,889 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>> (AgentManager-Handler-10:null) Seq 4-8064821032713715883: >>>>>>> Forwarding >>>>>>> Seq 4-8064821032713715883: { Cmd , MgmtId: 227448510156211, >>>>>>> via: 4, >>>>>>> Ver: v1, Flags: 100111, >>>>>>> >>>>>>> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.ap >>>>>>> >>>>>>> ache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/5 >>>>>>> >>>>>>> /af949612-838f-3a6d-931b-312e612db740.vhd","origUrl":"http://download.c >>>>>>> >>>>>>> loud.com/templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-30 >>>>>>> >>>>>>> 17-11e5-8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksu >>>>>>> >>>>>>> m":"905cec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS >>>>>>> >>>>>>> >>>>>>> 5.6(64-bit) no GUI >>>>>>> >>>>>>> (XenServer)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":" >>>>>>> >>>>>>> nfs://10.0.0.100/storage/secondary","_role":"Image"}},"name":"centos56- >>>>>>> >>>>>>> x86_64-xen","hypervisorType":"XenServer"}},"destTO":{"org.apache.clouds >>>>>>> >>>>>>> tack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com >>>>>>> >>>>>>> /templates/builtin/centos56-x86_64.vhd.bz2","uuid":"80b60e46-3017-11e5- >>>>>>> >>>>>>> 8736-00259091a13a","id":5,"format":"VHD","accountId":1,"checksum":"905c >>>>>>> >>>>>>> ec879afd9c9d22ecc8036131a180","hvm":false,"displayText":"CentOS >>>>>>> 5.6(64-bit) no GUI >>>>>>> >>>>>>> (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.Primar >>>>>>> >>>>>>> yDataStoreTO":{"uuid":"2df26406-31bf-3a95-8a61-f5008defd9a0","id":4,"po >>>>>>> >>>>>>> olType":"NetworkFilesystem","host":"10.0.0.100","path":"/storage/xen/pr >>>>>>> >>>>>>> imary","port":2049,"url":"NetworkFilesystem://10.0.0.100/storage/xen/pr >>>>>>> >>>>>>> imary/?ROLE=Primary&STOREUUID=2df26406-31bf-3a95-8a61-f5008defd9a0"}}," >>>>>>> >>>>>>> name":"centos56-x86_64-xen","hypervisorType":"XenServer"}},"executeInSe >>>>>>> >>>>>>> quence":true,"options":{},"wait":10800}}] >>>>>>> } to 116784073679673 >>>>>>> >>>>>>> >>>>>>> ----------------------------------------- >>>>>>> >>>>>>> Anyone have Any ideas? thanks. >>>>>>> >>>>>>> ----------- >>>>>>> Cao Tong >>>>>>> >>>>>>> On 07/21/2015 06:14 PM, tony_caot...@163.com wrote: >>>>>>>> Thanks all, >>>>>>>> >>>>>>>> I have already reinstall my hosts for preparing a new clear >>>>>>>> environment to restart my research. >>>>>>>> >>>>>>>> ----------- >>>>>>>> Cao Tong >>>>>>>> >>>>>>>> On 07/20/2015 09:24 PM, Prashant s wrote: >>>>>>>>> some questions : >>>>>>>>> >>>>>>>>> can you please tell ... >>>>>>>>> >>>>>>>>> 1. is your NFS storage or your primary Storage Repository in >>>>>>>>> connected >>>>>>>>> mode with no red cross mark on them in xencenter. >>>>>>>>> 2. did you change any passwords on the xenservers ? >>>>>>>>> 3. is the cloudstack web ui up , can you login to the cloudstack >>>>>>>>> web page. >>>>>>>>> 4. *are the zone , pod, or clusters in unmanaged or disabled >>>>>>>>> state >>>>>>>>> ? * >>>>>>>>> *5. is all the hosts in connected state ? * >>>>>>>>> *6. did you run yum update on host reboot on the cs manager >>>>>>>>> vm ? * >>>>>>>>> *7. system vms are stateless you can kill them and cs will >>>>>>>>> recreate a new >>>>>>>>> one .. so dont worry :-) * >>>>>>>>> >>>>>>>>> >>>>>>>>> *thanks * >>>>>>>>> *prashant * >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jul 20, 2015 at 3:47 AM, <tony_caot...@163.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, I restartd All hosts (one mgr and xenserver) again. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Following is the error log. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode] >>>>>>>>>> (StatsCollector-3:ctx-692a5392) Could not find exception: >>>>>>>>>> com.cloud.exception.OperationTimedoutException in error code >>>>>>>>>> list >>>>>>>>>> for >>>>>>>>>> exceptions >>>>>>>>>> 2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache] >>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed >>>>>>>>>> out on null >>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache] >>>>>>>>>> (StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: >>>>>>>>>> Cancelling. >>>>>>>>>> 2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl] >>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to send storage pool >>>>>>>>>> command to >>>>>>>>>> Pool[1|NetworkFilesystem] via 1 >>>>>>>>>> com.cloud.exception.OperationTimedoutException: Commands >>>>>>>>>> 3176445112179752972 to Host 1 timed out after 3600 >>>>>>>>>> at >>>>>>>>>> com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436) >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java: >>>>>>>>>> >>>>>>>>>> 433) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java: >>>>>>>>>> >>>>>>>>>> 362) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:1000) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:392) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:406) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC >>>>>>>>>> >>>>>>>>>> ollector.java:642) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M >>>>>>>>>> >>>>>>>>>> anagedContextRunnable.java:49) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c >>>>>>>>>> >>>>>>>>>> all(DefaultManagedContext.java:56) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal >>>>>>>>>> >>>>>>>>>> lWithContext(DefaultManagedContext.java:103) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run >>>>>>>>>> >>>>>>>>>> WithContext(DefaultManagedContext.java:53) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man >>>>>>>>>> >>>>>>>>>> agedContextRunnable.java:46) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4 >>>>>>>>>> >>>>>>>>>> 71) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >>>>>>>>>> >>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >>>>>>>>>> >>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor >>>>>>>>>> >>>>>>>>>> .java:1145) >>>>>>>>>> >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> 2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector] >>>>>>>>>> (StatsCollector-3:ctx-692a5392) Unable to reach >>>>>>>>>> Pool[1|NetworkFilesystem] >>>>>>>>>> com.cloud.exception.StorageUnavailableException: Resource >>>>>>>>>> [StoragePool:1] >>>>>>>>>> is unreachable: Unable to send command to the pool >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:1010) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:392) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.j >>>>>>>>>> >>>>>>>>>> ava:406) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsC >>>>>>>>>> >>>>>>>>>> ollector.java:642) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M >>>>>>>>>> >>>>>>>>>> anagedContextRunnable.java:49) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c >>>>>>>>>> >>>>>>>>>> all(DefaultManagedContext.java:56) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal >>>>>>>>>> >>>>>>>>>> lWithContext(DefaultManagedContext.java:103) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run >>>>>>>>>> >>>>>>>>>> WithContext(DefaultManagedContext.java:53) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man >>>>>>>>>> >>>>>>>>>> agedContextRunnable.java:46) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4 >>>>>>>>>> >>>>>>>>>> 71) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >>>>>>>>>> >>>>>>>>>> .access$301(ScheduledThreadPoolExecutor.java:178) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >>>>>>>>>> >>>>>>>>>> .run(ScheduledThreadPoolExecutor.java:293) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor >>>>>>>>>> >>>>>>>>>> .java:1145) >>>>>>>>>> >>>>>>>>>> at >>>>>>>>>> >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto >>>>>>>>>> >>>>>>>>>> r.java:615) >>>>>>>>>> >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> >>>>>>>>>> ----------- >>>>>>>>>> Cao Tong >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 07/20/2015 02:52 PM, tony_caot...@163.com wrote: >>>>>>>>>> >>>>>>>>>>> No, no one's IP was changed. >>>>>>>>>>> >>>>>>>>>>> 1. In xenserver I can not login systemvms using the internal IP >>>>>>>>>>> like >>>>>>>>>>> '169.254.1.112', There shoud be a bridge network for this >>>>>>>>>>> right? it is >>>>>>>>>>> gone. >>>>>>>>>>> >>>>>>>>>>> 2. I try to delete xenserver host from CS on web, it also >>>>>>>>>>> failed >>>>>>>>>>> with >>>>>>>>>>> lots of logs like following, then memory is full and mangement >>>>>>>>>>> down... >>>>>>>>>>> >>>>>>>>>>> 2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>>>>>> (AgentManager-Handler-15:null) Seq 1-7282039122481381399: >>>>>>>>>>> Forwarding Seq >>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, >>>>>>>>>>> via: 1, >>>>>>>>>>> Ver: v1, >>>>>>>>>>> Flags: 100111, >>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>>>>>>>> 192405008094602 >>>>>>>>>>> 2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>>>>>> (AgentManager-Handler-5:null) Seq 1-7282039122481381399: >>>>>>>>>>> Forwarding Seq >>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, >>>>>>>>>>> via: 1, >>>>>>>>>>> Ver: v1, >>>>>>>>>>> Flags: 100111, >>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>>>>>>>> 192405008094602 >>>>>>>>>>> 2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>>>>>> (AgentManager-Handler-1:null) Seq 1-7282039122481381399: >>>>>>>>>>> Forwarding Seq >>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, >>>>>>>>>>> via: 1, >>>>>>>>>>> Ver: v1, >>>>>>>>>>> Flags: 100111, >>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>>>>>>>> 192405008094602 >>>>>>>>>>> 2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>>>>>>>>> (AgentManager-Handler-14:null) Seq 1-7282039122481381399: >>>>>>>>>>> Forwarding Seq >>>>>>>>>>> 1-7282039122481381399: { Cmd , MgmtId: 104062526015411, >>>>>>>>>>> via: 1, >>>>>>>>>>> Ver: v1, >>>>>>>>>>> Flags: 100111, >>>>>>>>>>> [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}] } to >>>>>>>>>>> 192405008094602 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I guess that, is there some service or daemons working for CS >>>>>>>>>>> is not up >>>>>>>>>>> on Xenserver ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----------- >>>>>>>>>>> Cao Tong >>>>>>>>>>> On 07/20/2015 02:35 PM, Rajani Karuturi wrote: >>>>>>>>>>> >>>>>>>>>>>> Did the management server ip change? >>>>>>>>>>>> management server ip in the configuration table is used my >>>>>>>>>>>> systemvms. >>>>>>>>>>>> select * from configuration where name like 'host'; >>>>>>>>>>>> >>>>>>>>>>>> If it changed, correct the value in db and restart systemvms. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ~Rajani >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 20, 2015 at 11:56 AM,<tony_caot...@163.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>>> I shutdown my cs-manager and xenserver last weekend, And now >>>>>>>>>>>>> the ssvm >>>>>>>>>>>>> and cpvm is disconnect, thost two was runing on xenserver. so >>>>>>>>>>>>> What >>>>>>>>>>>>> should i do right now ? >>>>>>>>>>>>> Please anybody help me and thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> In xenserver I found that the three system VMs are not >>>>>>>>>>>>> running. >>>>>>>>>>>>> my xenserver seems can not reconnect to CS-manager. and it >>>>>>>>>>>>> seams not >>>>>>>>>>>>> under control of CS. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> What is the right steps of shutdown all CS group machines and >>>>>>>>>>>>> resume >>>>>>>>>>>>> them? >>>>>>>>>>>>> How can i let my xenserver reconnected ? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ----------- >>>>>>>>>>>>> Cao Tong >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> >> > >
