Hi Simon, I just noticed that from other Hosts I can't ssh into the SSVM:
[root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.161 ssh: connect to host 169.254.3.161 port 3922: No route to host I can only ssh from one Host: [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.161 Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 Thanks & Regards, Nezar. On 18 March 2016 at 10:36, Nezar Madbouh <nezar.madb...@ipscape.com.au> wrote: > Hi Simon, > > Yes existing VMs are successfully running. Yesterday I was able to create > a VM in Dev Environment but I can't create VMs in Prod environment. > > Thanks, > Nezar. > > On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote: > >> I'm assuming you have other VMs successfully running, correct? So you can >> write to your primary storage. >> >> >> >> ________________________________________ >> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> Sent: Thursday, March 17, 2016 4:43 PM >> To: users@cloudstack.apache.org >> Subject: Re: Unable to create a New VM - Error message: Unable to start >> instance due to Unable to get answer that is of class >> com.cloud.agent.api.StartAnswer >> >> Hi Simon and Shweta, >> >> Please see the result below: >> >> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >> ================================================ >> First DNS server is 8.8.8.8 >> PING 8.8.8.8 (8.8.8.8): 48 data bytes >> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms >> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms >> --- 8.8.8.8 ping statistics --- >> 2 packets transmitted, 2 packets received, 0% packet loss >> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms >> Good: Can ping DNS server >> ================================================ >> Good: DNS resolves download.cloud.com >> ================================================ >> nfs is currently mounted >> Mount point is /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 >> Good: Can write to mount point >> ================================================ >> Management server is 10.61.153.20. Checking connectivity. >> Good: Can connect to management server port 8250 >> ================================================ >> Good: Java process is running >> ================================================ >> Tests Complete. Look for ERROR or WARNING above. >> root@s-11-VM:~# >> >> Thanks & Regards, >> Nezar. >> >> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote: >> >> > Nezar, >> > >> > Log into the Cloudstack UI. >> > >> > Click on "Infrastructure" on the left menu. Click on View All below >> > "System VMs". find the item marked "Secondary Storage VM" and then >> click >> > on the Name of that VM on the left (it will be s-XXXX-VM). You'll see a >> > bunch of information about the VM. Find the Host line and also the Link >> > Local IP Address line and then follow the original instructions in my >> > previous email. >> > >> > - Si >> > >> > >> > >> > >> > >> > ________________________________________ >> > From: Shweta Agarwal <shweta.agarw...@accelerite.com> >> > Sent: Wednesday, March 16, 2016 12:16 AM >> > To: users@cloudstack.apache.org >> > Subject: Re: Unable to create a New VM - Error message: Unable to start >> > instance due to Unable to get answer that is of class >> > com.cloud.agent.api.StartAnswer >> > >> > Hi Nezar, >> > >> > You can find host name on detail view of SSVM in UI . >> > You can also use list api command listSystemVms to get the host details >> of >> > system vms. >> > >> > Thanks >> > Shweta >> > >> > >> > ________________________________________ >> > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > Sent: Wednesday, March 16, 2016 9:58 AM >> > To: users@cloudstack.apache.org >> > Subject: Re: Unable to create a New VM - Error message: Unable to start >> > instance due to Unable to get answer that is of class >> > com.cloud.agent.api.StartAnswer >> > >> > Hi Simon, >> > >> > How can I find out which host the SSVM is on? >> > >> > Thanks & Regards, >> > Nezar. >> > >> > On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> wrote: >> > >> > > Can you ssh into the Secondary Storage VM? >> > > >> > > This can be done by getting the link-local-ip address from the >> > > infrastructure menu in the UI and then sshing into the host the SSVM >> is >> > on >> > > and issuing this command as root: ssh -i /root/.ssh/id_rsa.cloud -p >> 3922 >> > > root@link-local-ip (replace link-local-ip with the link local ip). >> > > >> > > Try running the SSVM diangnostic script: >> > > usr/local/cloud/systemvm/ssvm-check.sh >> > > >> > > - Si >> > > ________________________________________ >> > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > Sent: Tuesday, March 15, 2016 7:49 PM >> > > To: users@cloudstack.apache.org >> > > Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> > > instance due to Unable to get answer that is of class >> > > com.cloud.agent.api.StartAnswer >> > > >> > > Hi Simon, >> > > >> > > Please see details below: >> > > >> > > >> > > mysql> use cloud; select id, name, status, type from host where id = >> 5; >> > > Reading table information for completion of table and column names >> > > You can turn off this feature to get a quicker startup with -A >> > > >> > > Database changed >> > > +----+------------------------------+--------+---------+ >> > > | id | name | status | type | >> > > +----+------------------------------+--------+---------+ >> > > | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | >> > > +----+------------------------------+--------+---------+ >> > > 1 row in set (0.00 sec) >> > > >> > > >> > > >> > > cat /var/log/cloudstack/agent/agent.log >> > > com.cloud.utils.exception.CloudRuntimeException: >> > > com.cloud.utils.exception.CloudRuntimeException: Can't find >> > > volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) >> > > at com.cloud.agent.Agent.processRequest(Agent.java:501) >> > > at >> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >> > > at com.cloud.utils.nio.Task.run(Task.java:84) >> > > at >> > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > > at >> > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:745) >> > > agent.log.2016-03-03.gz (END) >> > > >> > > >> > > >> > > [root@eqx-cs-cmp-02 agent]# virsh pool-list >> > > Name State Autostart >> > > ----------------------------------------- >> > > 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no >> > > b3647480-5709-36eb-9ddd-e2c666f30b57 active no >> > > e3831246-ff55-4af4-9964-a17565edaccc active no >> > > >> > > >> > > >> > > >> > > [Root @ eqx-cs-cmp-02 agent] # df -h >> > > Filesystem Size Used Avail Use% Mounted on >> > > /dev/mapper/vg_root-lv_root >> > > 203G 215G 1.3G 1% / >> > > tmpfs 48g 48g 0 0% / dev / shm >> > > / Dev / sda1 477m 427m 25M 6% / boot >> > > xx.xx.xx.xx:/Cloudstack/primary >> > > 7.6T 4.1T 3.5T 54% >> > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 >> > > >> > > >> > > >> > > >> > > I also noticed the following in cloudstack-agent.err >> > > >> > > libvirt: QEMU Driver error : Domain not found: no domain with matching >> > name >> > > 'i-11-600-VM' >> > > libvirt: Storage Driver error : Storage volume not found: no storage >> vol >> > > with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' >> > > >> > > Your help is much appreciated, >> > > Nezar >> > > >> > > >> > > On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> wrote: >> > > >> > > > Nezar, >> > > > >> > > > ok, I just took a look at a pastebin you posted. >> > > > >> > > > I see it complaining about the inability to find a volume prior to >> > > > throwing the host exception. >> > > > >> > > > Firstly, let's find out who host 5 is. Can you log into your MySQL >> > server >> > > > and run this: >> > > > >> > > > use cloud;select id,name,status,type from host where id=5; >> > > > >> > > > Can you post the results? >> > > > >> > > > If it's a host, can you post the /var/log/cloudstack/agent/agent.log >> > from >> > > > that host? >> > > > >> > > > And also run virsh pool-list on the host in question? >> > > > >> > > > - Si >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > ________________________________________ >> > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > Sent: Tuesday, March 15, 2016 6:11 PM >> > > > To: users@cloudstack.apache.org >> > > > Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> > > > instance due to Unable to get answer that is of class >> > > > com.cloud.agent.api.StartAnswer >> > > > >> > > > Hi Simon, >> > > > >> > > > When you say "double check your routers tied to the network", do you >> > mean >> > > > the physical routers or the virtual routers? >> > > > >> > > > When I try to create a VM in Prod Environment it fails. I can't ping >> > the >> > > > Prod virtual router from management server or Host. >> > > > >> > > > But I can ping Dev virtual router from management server and Host >> and I >> > > was >> > > > able to create a VM in Dev Environment. >> > > > >> > > > How can I troubleshoot the Prod Virtual router. I launched the >> console >> > > but >> > > > couldn't login. >> > > > >> > > > >> > > > Please note that we need to take into consideration that the >> affected >> > > > network is the Production network and we need to minimise service >> > outage >> > > as >> > > > much as possible. >> > > > >> > > > Your help is much appreciated, >> > > > Nezar >> > > > >> > > > >> > > > >> > > > On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> wrote: >> > > > >> > > > > I've seen this before where a Virtual Router has crashed and the >> > > > > management server can't communicate with it. >> > > > > Are you using an advanced network setup? >> > > > > >> > > > > If so, double check your routers tied to the network where you are >> > > > > attempting to start an instance. >> > > > > >> > > > > - Si >> > > > > >> > > > > >> > > > > ________________________________________ >> > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > > Sent: Tuesday, March 15, 2016 4:02 PM >> > > > > To: users@cloudstack.apache.org >> > > > > Subject: Re: Unable to create a New VM - Error message: Unable to >> > start >> > > > > instance due to Unable to get answer that is of class >> > > > > com.cloud.agent.api.StartAnswer >> > > > > >> > > > > Hi Glenn, >> > > > > >> > > > > We have 16 hosts, all KVM. >> > > > > >> > > > > The error is not specific to Host 5. Every time I try to create a >> VM >> > it >> > > > > gives me the same error message but on different host. >> > > > > >> > > > > How do I restart the agent on the host. Is it service affecting? >> > > > > >> > > > > Thanks very much for your help. >> > > > > Neza >> > > > > >> > > > > On 15 March 2016 at 17:37, Glenn Wagner < >> glenn.wag...@shapeblue.com> >> > > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Can you explain your environment first , how many hosts , what >> is >> > > your >> > > > > > hypervisors (KVM,Xenserver,VMware) >> > > > > > >> > > > > > The error seems to be with your Host 5 Hypervisor >> > > > > > Resource [Host:5] is unreachable: >> > > > > > >> > > > > > Go to the infrastructure section and see what host5 is >> reporting, >> > > > > > Have you tried to restart the agent on the Host? >> > > > > > >> > > > > > Thanks >> > > > > > Glenn >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > [image: ShapeBlue] <http://www.shapeblue.com> >> > > > > > Glenn Wagner >> > > > > > Senior Consultant , ShapeBlue >> > > > > > d: * | s: +27 21 527 0091* >> <%7C%20s:%20+27%2021%20527%200091> | >> > > m: >> > > > > > *+27 73 917 4111* <+27%2073%20917%204111> >> > > > > > e: *glenn.wag...@shapeblue.com | t: * >> > > > > > <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * >> www.shapeblue.com >> > * >> > > > > > <http://www.shapeblue.com> >> > > > > > a: >> > > > > > 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town >> > 7130 >> > > > > South Africa >> > > > > > Shape Blue Ltd is a company incorporated in England & Wales. >> > > ShapeBlue >> > > > > > Services India LLP is a company incorporated in India and is >> > operated >> > > > > under >> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda >> is >> > a >> > > > > > company incorporated in Brasil and is operated under license >> from >> > > Shape >> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The >> > > Republic >> > > > of >> > > > > > South Africa and is traded under license from Shape Blue Ltd. >> > > ShapeBlue >> > > > > is >> > > > > > a registered trademark. >> > > > > > This email and any attachments to it may be confidential and are >> > > > intended >> > > > > > solely for the use of the individual to whom it is addressed. >> Any >> > > views >> > > > > or >> > > > > > opinions expressed are solely those of the author and do not >> > > > necessarily >> > > > > > represent those of Shape Blue Ltd or related companies. If you >> are >> > > not >> > > > > the >> > > > > > intended recipient of this email, you must neither take any >> action >> > > > based >> > > > > > upon its contents, nor copy or show it to anyone. Please contact >> > the >> > > > > sender >> > > > > > if you believe you have received this email in error. >> > > > > > >> > > > > > >> > > > > > -----Original Message----- >> > > > > > From: Nezar Madbouh [mailto:nezar.madb...@ipscape.com.au] >> > > > > > Sent: Tuesday, 15 March 2016 5:45 AM >> > > > > > To: users@cloudstack.apache.org >> > > > > > Subject: Re: Unable to create a New VM - Error message: Unable >> to >> > > start >> > > > > > instance due to Unable to get answer that is of class >> > > > > > com.cloud.agent.api.StartAnswer >> > > > > > >> > > > > > Hi Jeremy, >> > > > > > >> > > > > > I have uploaded the contents to http://pastebin.com/1J3mmgTh >> > > > > > >> > > > > > your help is much appreciated, >> > > > > > Nezar. >> > > > > > >> > > > > > On 15 March 2016 at 13:46, Jeremy Peterson < >> jpeter...@acentek.net> >> > > > > wrote: >> > > > > > >> > > > > > > I don't see any attachments can you create a pastebin. That's >> > > easier >> > > > > > > to look through as I think attachments are removed before >> being >> > > sent >> > > > > > > to user group. >> > > > > > > >> > > > > > > Jeremy >> > > > > > > >> > > > > > > >> > > > > > > -------- Original message -------- >> > > > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > > > > Date: 3/14/2016 6:38 PM (GMT-06:00) >> > > > > > > To: users@cloudstack.apache.org >> > > > > > > Subject: Unable to create a New VM - Error message: Unable to >> > start >> > > > > > > instance due to Unable to get answer that is of class >> > > > > > > com.cloud.agent.api.StartAnswer >> > > > > > > >> > > > > > > Hi Shweta and Sadhu, >> > > > > > > >> > > > > > > Please find attached managerment-server.log which contains the >> > logs >> > > > > > > since I started creating a new VM (VM id 600). >> > > > > > > >> > > > > > > Please note that we don't have agent.log files >> > > > > > > >> > > > > > > Your help is much appreciated. >> > > > > > > >> > > > > > > Regards, >> > > > > > > Nezar. >> > > > > > > >> > > > > > Find out more about ShapeBlue and our range of CloudStack >> related >> > > > > services: >> > > > > > IaaS Cloud Design & Build >> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge >> – >> > > rapid >> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/> >> > > > > > CloudStack Consulting < >> > http://shapeblue.com/cloudstack-consultancy/> >> > > | >> > > > > CloudStack >> > > > > > Software Engineering >> > > > > > <http://shapeblue.com/cloudstack-software-engineering/> >> > > > > > CloudStack Infrastructure Support >> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> | >> > > CloudStack >> > > > > > Bootcamp Training Courses < >> > http://shapeblue.com/cloudstack-training/ >> > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > >> > DISCLAIMER >> > ========== >> > This e-mail may contain privileged and confidential information which is >> > the property of Accelerite, a Persistent Systems business. It is >> intended >> > only for the use of the individual or entity to which it is addressed. >> If >> > you are not the intended recipient, you are not authorized to read, >> retain, >> > copy, print, distribute or use this message. If you have received this >> > communication in error, please notify the sender and delete all copies >> of >> > this message. Accelerite, a Persistent Systems business does not accept >> any >> > liability for virus infected mails. >> > >> > >
