Here is what i need to understand, if you are using SharedMountPoint, then /mnt/storage must be preset on all KVM hosts in a persistent manner (i.e. fstab).
However, if you are using NFS, you can just let cloudstack manage the NFS mount points by adding the storage as NFS type. On 4/15/16 12:46 AM, ilya wrote: > Can you goto primary storage in cloudstack UI, is this what you have > defined for your NFS Server? > > "SharedMountPoint","host":"localhost","path":"/mnt/storage" > > If so, please use IP or FQDN instead.. > > > On 4/14/16 10:39 PM, Nezar Madbouh wrote: >> Hi Ilya, >> >> Here is the output of mount command: >> >> [root@eqx-cs-cmp-01 ~]# mount >> /dev/mapper/vg_root-lv_root on / type ext4 (rw) >> proc on /proc type proc (rw) >> sysfs on /sys type sysfs (rw) >> devpts on /dev/pts type devpts (rw,gid=5,mode=620) >> tmpfs on /dev/shm type tmpfs (rw) >> /dev/sda1 on /boot type ext4 (rw) >> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) >> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) >> nfsd on /proc/fs/nfsd type nfsd (rw) >> 10.61.154.10:/Cloudstack/primary on >> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs >> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) >> [root@eqx-cs-cmp-01 ~]# >> >> >> On 15 April 2016 at 15:22, Nezar Madbouh <nezar.madb...@ipscape.com.au> >> wrote: >> >>> Hi Ilya, >>> >>> Thanks for the instructions. I followed them and pasted the information on >>> pastebin: >>> >>> management-server logs job-14938 >>> http://pastebin.com/GAiy8u5H >>> >>> Agent logs >>> http://pastebin.com/mfiADsJc >>> >>> >>> Your help is much appreciated. >>> >>> Nezar. >>> >>> On 15 April 2016 at 13:58, ilya <ilya.mailing.li...@gmail.com> wrote: >>> >>>> Here is how i would troubleshoot this: >>>> >>>> 1) ssh to MS host, run >>>> >>>> tail -f /var/cloudstack/management/management-server.log | grep -i >>>> StartVMCmdByAdmin >>>> >>>> 2) on cloudstack UI, press start on the VM and watch the tail log >>>> >>>> 3) on the tail log, you should see a StartVirtualMachine api call show >>>> up with job ID. For example: >>>> >>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO >>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine, >>>> instanceId: 7648, cmd: >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo: >>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"}, >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated: >>>> null, lastPolled: null, created: null} >>>> >>>> Each task in cloudstack spins up its own job, in this case its >>>> *job-291575* >>>> >>>> 4) now execute >>>>> grep -A3 -B3 job-291575 >>>> /var/cloudstack/management/management-server.log | tee job-291575.log >>>>> grep job-291575 /var/cloudstack/management/management-server.log | grep >>>> via | head -1 >>>> >>>> you should see a response like so, >>>> >>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache] >>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq >>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd , >>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1, >>>> Flags: 100011, >>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238"," >>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893 >>>> >>>> We are looking for this line "via: 441(my-kvm-host01.example.com)" >>>> >>>> SSH to my-kvm-host01.example.com, and copy the >>>> /var/log/cloudstack/agent.log to pastebin >>>> >>>> also copy job-291575.log to pastebin >>>> >>>> I'll try to have a look, though when i get chance. >>>> >>>> >>>> Also paste the output of mount command on kvm host in question. I'm >>>> suspecting one of your storage pools maybe bad. >>>> >>>> Lastly, what distro are you running for KVM? >>>> >>>> Regards >>>> ilya >>>> >>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote: >>>>> Hi Ilya, >>>>> >>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod >>>>> Virtual router to be deleted automatically because no VM is running on >>>> this >>>>> vlan but it wasn't deleted. >>>>> >>>>> Am I right to expect the VR to be deleted automatically if no VM is >>>> running >>>>> in that vlan? >>>>> >>>>> Thanks, >>>>> Nezar. >>>>> >>>>> On 15 April 2016 at 09:53, Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>> wrote: >>>>> >>>>>> Hi Ilya, >>>>>> >>>>>> I have run the script and it didn't return any results: >>>>>> >>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh >>>>>> [root@eqx-cs-mgt-01 ~]# >>>>>> >>>>>> >>>>>> I also ran the command inside mysql: >>>>>> >>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id, >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like >>>>>> "pseudoJobDispatcher"; >>>>>> Empty set (0.00 sec) >>>>>> >>>>>> your help is much appreciated, >>>>>> Nezar. >>>>>> >>>>>> >>>>>> On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> wrote: >>>>>> >>>>>>> It appears you have pending jobs that are not letting you go through. >>>>>>> >>>>>>> Usually stop and starting management service should be enough. >>>>>>> >>>>>>> However, try running this script from your management server to show >>>> us >>>>>>> what you have running in async_job table. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> #!/bin/bash >>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S) >>>>>>>> DBPASS=$(java -classpath >>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar >>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat >>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password | >>>> awk >>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat >>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1) >>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep >>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 ) >>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep >>>>>>> db.cloud.username | awk -F'=' '{print $2}') >>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep >>>> db.cloud.name >>>>>>> | awk -F'=' '{print $2}') >>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep >>>>>>> db.cloud.port | awk -F'=' '{print $2}') >>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB" >>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id, >>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM >>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like >>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote: >>>>>>>> Hi Simon, >>>>>>>> >>>>>>>> The following work was done but unfortunately the problem is not >>>> fixed: >>>>>>>> >>>>>>>> >>>>>>>> on management server: >>>>>>>> >>>>>>>> service cloudstack-management stop >>>>>>>> >>>>>>>> >>>>>>>> on the host: >>>>>>>> >>>>>>>> service cloudstack-agent stop >>>>>>>> >>>>>>>> service libvirtd restart >>>>>>>> >>>>>>>> service cloudstack-agent start >>>>>>>> >>>>>>>> >>>>>>>> on management server: >>>>>>>> >>>>>>>> service cloudstack-management start >>>>>>>> >>>>>>>> Regards, >>>>>>>> Nezar. >>>>>>>> >>>>>>>> On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote: >>>>>>>> >>>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed >>>>>>>>> shutdown before moving on to the next step. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________________ >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM >>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>> start >>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>> >>>>>>>>> Hi Simon, >>>>>>>>> >>>>>>>>> Can you please check if the following sequence of action is correct: >>>>>>>>> >>>>>>>>> on management server: >>>>>>>>> >>>>>>>>> service cloudstack-management stop >>>>>>>>> >>>>>>>>> >>>>>>>>> on the host: >>>>>>>>> >>>>>>>>> service cloudstack-agent stop >>>>>>>>> >>>>>>>>> service libvirtd restart >>>>>>>>> >>>>>>>>> service cloudstack-agent start >>>>>>>>> >>>>>>>>> >>>>>>>>> on management server: >>>>>>>>> >>>>>>>>> service cloudstack-management start >>>>>>>>> >>>>>>>>> Thanks very much for your help. >>>>>>>>> Nezar. >>>>>>>>> >>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote: >>>>>>>>> >>>>>>>>>> Shutting down all the management services should clear the jobs. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ________________________________________ >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM >>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>> start >>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>> >>>>>>>>>> Hi Simon, >>>>>>>>>> >>>>>>>>>> I will plan to restart agents, libvirtd and management services. >>>>>>>>>> >>>>>>>>>> How do I clear hung jobs. >>>>>>>>>> >>>>>>>>>> Your help is much appreciated, >>>>>>>>>> Nezar. >>>>>>>>>> >>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> wrote: >>>>>>>>>> >>>>>>>>>>> Nezar, >>>>>>>>>>> >>>>>>>>>>> I've looked through the code related to your release and it seems >>>>>>> that >>>>>>>>> in >>>>>>>>>>> order to throw this exception, 10 attempts have been made to >>>> retrieve >>>>>>>>> the >>>>>>>>>>> disk information prior to the exception being thrown. >>>>>>>>>>> >>>>>>>>>>> It appears there is something really odd going on here. I'm >>>> starting >>>>>>>>> to >>>>>>>>>>> wonder whether it might be worth while for you to stop all of your >>>>>>>>>> agents, >>>>>>>>>>> plus management server processes, restart libvirtd on each of the >>>>>>> hosts >>>>>>>>>> and >>>>>>>>>>> then bring the agents back up prior to starting the management >>>>>>> servers. >>>>>>>>>>> >>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to >>>>>>>>> indicate >>>>>>>>>>> you have some jobs hung, for example: >>>>>>>>>>> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for >>>> 1660851 >>>>>>>>>> seconds >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for >>>> 4253003 >>>>>>>>>> seconds >>>>>>>>>>> >>>>>>>>>>> Restarting the agents, libvirtd and management will not cause an >>>>>>>>> outage >>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you >>>>>>>>> should >>>>>>>>>>> complete it in a maintenance window. >>>>>>>>>>> >>>>>>>>>>> - Si >>>>>>>>>>> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM >>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>> start >>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>> >>>>>>>>>>> Hi Simon, >>>>>>>>>>> >>>>>>>>>>> Please see answers to your questions below. I am not sure if I >>>>>>> provided >>>>>>>>>> the >>>>>>>>>>> correct answer for the primary and secondary storage >>>> configuration. >>>>>>>>>> Please >>>>>>>>>>> let me know if this is not what you are after: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage >>>>>>>>>> configuration >>>>>>>>>>> and your primary storage configuration from the UI with the >>>> details? >>>>>>>>>>> >>>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G >>>>>>>>>>> *************************** 1. row *************************** >>>>>>>>>>> category: Account Defaults >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: max.account.secondary.storage >>>>>>>>>>> value: 400 >>>>>>>>>>> description: The default maximum secondary storage space (in >>>> GiB) >>>>>>>>> that >>>>>>>>>>> can be used for an account >>>>>>>>>>> default_value: 400 >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 2. row *************************** >>>>>>>>>>> category: Project Defaults >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: max.project.secondary.storage >>>>>>>>>>> value: 400 >>>>>>>>>>> description: The default maximum secondary storage space (in >>>> GiB) >>>>>>>>> that >>>>>>>>>>> can be used for a project >>>>>>>>>>> default_value: 400 >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 3. row *************************** >>>>>>>>>>> category: Hidden >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: secondary.storage.vm >>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw== >>>>>>>>>>> description: Deploys a VM per zone to manage secondary storage >>>> if >>>>>>>>> true, >>>>>>>>>>> otherwise secondary storage is mounted on management server >>>>>>>>>>> default_value: NULL >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 4. row *************************** >>>>>>>>>>> category: Advanced >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: ExtensionRegistry >>>>>>>>>>> name: secondary.storage.vm.allocators.exclude >>>>>>>>>>> value: NULL >>>>>>>>>>> description: Extensions to exclude from being registered >>>>>>>>>>> default_value: NULL >>>>>>>>>>> updated: 2014-12-04 00:18:15 >>>>>>>>>>> scope: Global >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 5. row *************************** >>>>>>>>>>> category: NetworkManager >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses >>>>>>>>>>> value: NULL >>>>>>>>>>> description: Specify the number of secondary ip addresses per >>>> nic >>>>>>> per >>>>>>>>>> vm >>>>>>>>>>> default_value: 256 >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> 5 rows in set (0.00 sec) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G >>>>>>>>>>> *************************** 1. row *************************** >>>>>>>>>>> category: Account Defaults >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: max.account.primary.storage >>>>>>>>>>> value: 200 >>>>>>>>>>> description: The default maximum primary storage space (in GiB) >>>>>>> that >>>>>>>>>> can >>>>>>>>>>> be used for an account >>>>>>>>>>> default_value: 200 >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 2. row *************************** >>>>>>>>>>> category: Project Defaults >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: management-server >>>>>>>>>>> name: max.project.primary.storage >>>>>>>>>>> value: 200 >>>>>>>>>>> description: The default maximum primary storage space (in GiB) >>>>>>> that >>>>>>>>>> can >>>>>>>>>>> be used for a project >>>>>>>>>>> default_value: 200 >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> *************************** 3. row *************************** >>>>>>>>>>> category: Storage >>>>>>>>>>> instance: DEFAULT >>>>>>>>>>> component: TemplateManager >>>>>>>>>>> name: primary.storage.download.wait >>>>>>>>>>> value: 10800 >>>>>>>>>>> description: In second, timeout for download template to primary >>>>>>>>>> storage >>>>>>>>>>> default_value: NULL >>>>>>>>>>> updated: NULL >>>>>>>>>>> scope: NULL >>>>>>>>>>> is_dynamic: 0 >>>>>>>>>>> 3 rows in set (0.00 sec) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Also, what is the Cloudstack version you are running and what is >>>> the >>>>>>> OS >>>>>>>>>>> you're running it on? >>>>>>>>>>> >>>>>>>>>>> Cloudstack 4.4.1 >>>>>>>>>>> >>>>>>>>>>> Centos 6.6 >>>>>>>>>>> >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line? >>>>>>>>>>> >>>>>>>>>>> Primary storage on Hypervisors has the following parameters >>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on >>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs >>>>>>>>>>> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Secondary storage on SSVM has the following parameters >>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on >>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans= >>>>>>> 2147483647 >>>>>>> >>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks & Regards, >>>>>>>>>>> Nezar. >>>>>>>>>>> >>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is >>>> the >>>>>>>>> OS >>>>>>>>>>>> you're running it on? >>>>>>>>>>>> >>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line? >>>>>>>>>>>> >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> From: Simon Weller <swel...@ena.com> >>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM >>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>>>> start >>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>> >>>>>>>>>>>> Nexar, >>>>>>>>>>>> >>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on >>>> the >>>>>>>>> host >>>>>>>>>>>> where the VM resides. >>>>>>>>>>>> >>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage >>>>>>>>>>> configuration >>>>>>>>>>>> and your primary storage configuration from the UI with the >>>> details? >>>>>>>>>>>> >>>>>>>>>>>> - Si >>>>>>>>>>>> >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM >>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>>>> start >>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>> >>>>>>>>>>>> Hi Simon, >>>>>>>>>>>> >>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM: >>>>>>>>>>>> >>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >>>>>>>>>>>> root@169.254.3.161 >>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I can only ssh from one Host: >>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >>>>>>>>>>>> root@169.254.3.161 >>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 >>>>>>>>>>>> >>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>> Nezar. >>>>>>>>>>>> >>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh < >>>>>>>>> nezar.madb...@ipscape.com.au >>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>> >>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able >>>> to >>>>>>>>>>> create >>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod >>>> environment. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Nezar. >>>>>>>>>>>>> >>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> >>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct? >>>> So >>>>>>>>>> you >>>>>>>>>>>> can >>>>>>>>>>>>>> write to your primary storage. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM >>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>>> to >>>>>>>>>>> start >>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Simon and Shweta, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please see the result below: >>>>>>>>>>>>>> >>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> First DNS server is 8.8.8.8 >>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms >>>>>>>>>>>>>> --- 8.8.8.8 ping statistics --- >>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss >>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms >>>>>>>>>>>>>> Good: Can ping DNS server >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> Good: DNS resolves download.cloud.com >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> nfs is currently mounted >>>>>>>>>>>>>> Mount point is >>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 >>>>>>>>>>>>>> Good: Can write to mount point >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity. >>>>>>>>>>>>>> Good: Can connect to management server port 8250 >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> Good: Java process is running >>>>>>>>>>>>>> ================================================ >>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above. >>>>>>>>>>>>>> root@s-11-VM:~# >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> >>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Nezar, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Log into the Cloudstack UI. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All >>>>>>>>>> below >>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and >>>>>>>>>> then >>>>>>>>>>>>>> click >>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM). >>>>>>>>> You'll >>>>>>>>>>> see >>>>>>>>>>>> a >>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also >>>>>>>>> the >>>>>>>>>>>> Link >>>>>>>>>>>>>>> Local IP Address line and then follow the original >>>> instructions >>>>>>>>> in >>>>>>>>>>> my >>>>>>>>>>>>>>> previous email. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM >>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>>>>>>>> to >>>>>>>>>>>> start >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Nezar, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI . >>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the >>>> host >>>>>>>>>>>> details >>>>>>>>>>>>>> of >>>>>>>>>>>>>>> system vms. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Shweta >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM >>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>>>>>>>> to >>>>>>>>>>>> start >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> How can I find out which host the SSVM is on? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> >>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from >>>> the >>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host >>>>>>>>> the >>>>>>>>>>> SSVM >>>>>>>>>>>>>> is >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>> and issuing this command as root: ssh -i >>>>>>>>> /root/.ssh/id_rsa.cloud >>>>>>>>>>> -p >>>>>>>>>>>>>> 3922 >>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link >>>>>>>>> local >>>>>>>>>>> ip). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Try running the SSVM diangnostic script: >>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>> Unable >>>>>>>>>> to >>>>>>>>>>>>>> start >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please see details below: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host >>>>>>>>> where >>>>>>>>>>> id = >>>>>>>>>>>>>> 5; >>>>>>>>>>>>>>>> Reading table information for completion of table and column >>>>>>>>>> names >>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with >>>> -A >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Database changed >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>>> | id | name | status | type | >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>>> 1 row in set (0.00 sec) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) >>>>>>>>>>>>>>>> at >>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list >>>>>>>>>>>>>>>> Name State Autostart >>>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no >>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no >>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h >>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root >>>>>>>>>>>>>>>> 203G 215G 1.3G 1% / >>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm >>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot >>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary >>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54% >>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with >>>>>>>>>>>> matching >>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>> 'i-11-600-VM' >>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no >>>>>>>>>>> storage >>>>>>>>>>>>>> vol >>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Your help is much appreciated, >>>>>>>>>>>>>>>> Nezar >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Nezar, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume >>>>>>>>>> prior >>>>>>>>>>> to >>>>>>>>>>>>>>>>> throwing the host exception. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your >>>>>>>>>>> MySQL >>>>>>>>>>>>>>> server >>>>>>>>>>>>>>>>> and run this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you post the results? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If it's a host, can you post the >>>>>>>>>>>> /var/log/cloudstack/agent/agent.log >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> that host? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>>> Unable >>>>>>>>>>> to >>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When you say "double check your routers tied to the >>>>>>>>> network", >>>>>>>>>> do >>>>>>>>>>>> you >>>>>>>>>>>>>>> mean >>>>>>>>>>>>>>>>> the physical routers or the virtual routers? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I >>>>>>>>>> can't >>>>>>>>>>>> ping >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> Prod virtual router from management server or Host. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and >>>>>>>>>>> Host >>>>>>>>>>>>>> and I >>>>>>>>>>>>>>>> was >>>>>>>>>>>>>>>>> able to create a VM in Dev Environment. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched >>>>>>>>> the >>>>>>>>>>>>>> console >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>> couldn't login. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please note that we need to take into consideration that the >>>>>>>>>>>>>> affected >>>>>>>>>>>>>>>>> network is the Production network and we need to minimise >>>>>>>>>>> service >>>>>>>>>>>>>>> outage >>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>> much as possible. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Your help is much appreciated, >>>>>>>>>>>>>>>>> Nezar >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed >>>>>>>>> and >>>>>>>>>>> the >>>>>>>>>>>>>>>>>> management server can't communicate with it. >>>>>>>>>>>>>>>>>> Are you using an advanced network setup? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where >>>>>>>>>> you >>>>>>>>>>>> are >>>>>>>>>>>>>>>>>> attempting to start an instance. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>>>> Unable >>>>>>>>>>>> to >>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Glenn, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to >>>>>>>>>>> create >>>>>>>>>>>> a >>>>>>>>>>>>>> VM >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> gives me the same error message but on different host. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service >>>>>>>>>>> affecting? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks very much for your help. >>>>>>>>>>>>>>>>>> Neza >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner < >>>>>>>>>>>>>> glenn.wag...@shapeblue.com> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts >>>>>>>>> , >>>>>>>>>>> what >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor >>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is >>>>>>>>>>>>>> reporting, >>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>> Glenn >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com> >>>>>>>>>>>>>>>>>>> Glenn Wagner >>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue >>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091* >>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> | >>>>>>>>>>>>>>>> m: >>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111> >>>>>>>>>>>>>>>>>>> e: *glenn.wag...@shapeblue.com | t: * >>>>>>>>>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * >>>>>>>>>>>>>> www.shapeblue.com >>>>>>>>>>>>>>> * >>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com> >>>>>>>>>>>>>>>>>>> a: >>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West >>>>>>>>>> Cape >>>>>>>>>>>> Town >>>>>>>>>>>>>>> 7130 >>>>>>>>>>>>>>>>>> South Africa >>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England & >>>>>>>>>> Wales. >>>>>>>>>>>>>>>> ShapeBlue >>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India >>>>>>>>> and >>>>>>>>>> is >>>>>>>>>>>>>>> operated >>>>>>>>>>>>>>>>>> under >>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil >>>>>>>>> Consultoria >>>>>>>>>>>> Ltda >>>>>>>>>>>>>> is >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under >>>>>>>>>> license >>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> Shape >>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered >>>>>>>>> by >>>>>>>>>>> The >>>>>>>>>>>>>>>> Republic >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue >>>>>>>>>>> Ltd. >>>>>>>>>>>>>>>> ShapeBlue >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> a registered trademark. >>>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential >>>>>>>>>> and >>>>>>>>>>>> are >>>>>>>>>>>>>>>>> intended >>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is >>>>>>>>>>> addressed. >>>>>>>>>>>>>> Any >>>>>>>>>>>>>>>> views >>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do >>>>>>>>>> not >>>>>>>>>>>>>>>>> necessarily >>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies. >>>>>>>>> If >>>>>>>>>>> you >>>>>>>>>>>>>> are >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take >>>>>>>>>> any >>>>>>>>>>>>>> action >>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please >>>>>>>>>>>> contact >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> sender >>>>>>>>>>>>>>>>>>> if you believe you have received this email in error. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto: >>>>>>>>> nezar.madb...@ipscape.com.au] >>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>>>>> Unable >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Jeremy, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have uploaded the contents to >>>>>>>>>>> http://pastebin.com/1J3mmgTh >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> your help is much appreciated, >>>>>>>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson < >>>>>>>>>>>>>> jpeter...@acentek.net> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin. >>>>>>>>>>>> That's >>>>>>>>>>>>>>>> easier >>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed >>>>>>>>>> before >>>>>>>>>>>>>> being >>>>>>>>>>>>>>>> sent >>>>>>>>>>>>>>>>>>>> to user group. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Jeremy >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -------- Original message -------- >>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00) >>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message: >>>>>>>>>> Unable >>>>>>>>>>>> to >>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which >>>>>>>>>> contains >>>>>>>>>>>> the >>>>>>>>>>>>>>> logs >>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Your help is much appreciated. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of >>>>>>>>> CloudStack >>>>>>>>>>>>>> related >>>>>>>>>>>>>>>>>> services: >>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | >>>>>>>>>>>> CSForge >>>>>>>>>>>>>> - >>>>>>>>>>>>>>>> rapid >>>>>>>>>>>>>>>>>>> IaaS deployment framework < >>>>>>>>> http://shapeblue.com/csforge/> >>>>>>>>>>>>>>>>>>> CloudStack Consulting < >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/> >>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>> CloudStack >>>>>>>>>>>>>>>>>>> Software Engineering >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/> >>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support >>>>>>>>>>>>>>>>>>> < >>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/> >>>>>>>>>> | >>>>>>>>>>>>>>>> CloudStack >>>>>>>>>>>>>>>>>>> Bootcamp Training Courses < >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DISCLAIMER >>>>>>>>>>>>>>> ========== >>>>>>>>>>>>>>> This e-mail may contain privileged and confidential >>>> information >>>>>>>>>>> which >>>>>>>>>>>> is >>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It >>>> is >>>>>>>>>>>>>> intended >>>>>>>>>>>>>>> only for the use of the individual or entity to which it is >>>>>>>>>>> addressed. >>>>>>>>>>>>>> If >>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to >>>>>>>>>> read, >>>>>>>>>>>>>> retain, >>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have >>>>>>>>> received >>>>>>>>>>> this >>>>>>>>>>>>>>> communication in error, please notify the sender and delete >>>> all >>>>>>>>>>> copies >>>>>>>>>>>>>> of >>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does >>>> not >>>>>>>>>>>> accept >>>>>>>>>>>>>> any >>>>>>>>>>>>>>> liability for virus infected mails. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>
