Hi Ilya, Please see answers to your questions below:
Primary Storage is configured to use local disk on the Hosts: Below are examples of 2 entries in "Infrastructure --> Primary Storage" in UI. Name: eqx-cs-cmp-01.ipscape.com.au Local storage Server: 10.61.153.22 Path: /var/lib/libvirtd/images (this is local disk on the host) cluster: eqx-cs-cluster-01 Scope: Host Name: eqx-cs-cmp-02.ipscape.com.au Local storage Server: 10.61.153.23 Path: /var/lib/libvirtd/images (this is local disk on the host) cluster: eqx-cs-cluster-01 Scope: Host and so on for all 16 Hosts Agent logs for eqx-cs-cmp-12 ------------------------------------------- logs are pasted in http://pastebin.com/tiMMhEzW Mount points: -------------------- Below are mount points from 2 Hosts: [root@eqx-cs-cmp-12 agent]# mount /dev/mapper/vg_root-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) 10.61.154.10:/Cloudstack/primary on /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,noac,actimeo=0,addr=10.61.154.10) [root@eqx-cs-cmp-01 ~]# mount /dev/mapper/vg_root-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) 10.61.154.10:/Cloudstack/primary on /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) Your help is much appreciated, Nezar. On 16 April 2016 at 03:32, ilya <ilya.mailing.li...@gmail.com> wrote: > Here is what i need to understand, if you are using SharedMountPoint, > then /mnt/storage must be preset on all KVM hosts in a persistent manner > (i.e. fstab). > > However, if you are using NFS, you can just let cloudstack manage the > NFS mount points by adding the storage as NFS type. > > On 4/15/16 12:46 AM, ilya wrote: > > Can you goto primary storage in cloudstack UI, is this what you have > > defined for your NFS Server? > > > > "SharedMountPoint","host":"localhost","path":"/mnt/storage" > > > > If so, please use IP or FQDN instead.. > > > > > > On 4/14/16 10:39 PM, Nezar Madbouh wrote: > >> Hi Ilya, > >> > >> Here is the output of mount command: > >> > >> [root@eqx-cs-cmp-01 ~]# mount > >> /dev/mapper/vg_root-lv_root on / type ext4 (rw) > >> proc on /proc type proc (rw) > >> sysfs on /sys type sysfs (rw) > >> devpts on /dev/pts type devpts (rw,gid=5,mode=620) > >> tmpfs on /dev/shm type tmpfs (rw) > >> /dev/sda1 on /boot type ext4 (rw) > >> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > >> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) > >> nfsd on /proc/fs/nfsd type nfsd (rw) > >> 10.61.154.10:/Cloudstack/primary on > >> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs > >> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) > >> [root@eqx-cs-cmp-01 ~]# > >> > >> > >> On 15 April 2016 at 15:22, Nezar Madbouh <nezar.madb...@ipscape.com.au> > >> wrote: > >> > >>> Hi Ilya, > >>> > >>> Thanks for the instructions. I followed them and pasted the > information on > >>> pastebin: > >>> > >>> management-server logs job-14938 > >>> http://pastebin.com/GAiy8u5H > >>> > >>> Agent logs > >>> http://pastebin.com/mfiADsJc > >>> > >>> > >>> Your help is much appreciated. > >>> > >>> Nezar. > >>> > >>> On 15 April 2016 at 13:58, ilya <ilya.mailing.li...@gmail.com> wrote: > >>> > >>>> Here is how i would troubleshoot this: > >>>> > >>>> 1) ssh to MS host, run > >>>> > >>>> tail -f /var/cloudstack/management/management-server.log | grep -i > >>>> StartVMCmdByAdmin > >>>> > >>>> 2) on cloudstack UI, press start on the VM and watch the tail log > >>>> > >>>> 3) on the tail log, you should see a StartVirtualMachine api call show > >>>> up with job ID. For example: > >>>> > >>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO > >>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine, > >>>> instanceId: 7648, cmd: > >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo: > >>>> > {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"}, > >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > >>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated: > >>>> null, lastPolled: null, created: null} > >>>> > >>>> Each task in cloudstack spins up its own job, in this case its > >>>> *job-291575* > >>>> > >>>> 4) now execute > >>>>> grep -A3 -B3 job-291575 > >>>> /var/cloudstack/management/management-server.log | tee job-291575.log > >>>>> grep job-291575 /var/cloudstack/management/management-server.log | > grep > >>>> via | head -1 > >>>> > >>>> you should see a response like so, > >>>> > >>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache] > >>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 > ctx-0988152e) Seq > >>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { > Cmd , > >>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1, > >>>> Flags: 100011, > >>>> > [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238"," > >>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893 > >>>> > >>>> We are looking for this line "via: 441(my-kvm-host01.example.com)" > >>>> > >>>> SSH to my-kvm-host01.example.com, and copy the > >>>> /var/log/cloudstack/agent.log to pastebin > >>>> > >>>> also copy job-291575.log to pastebin > >>>> > >>>> I'll try to have a look, though when i get chance. > >>>> > >>>> > >>>> Also paste the output of mount command on kvm host in question. I'm > >>>> suspecting one of your storage pools maybe bad. > >>>> > >>>> Lastly, what distro are you running for KVM? > >>>> > >>>> Regards > >>>> ilya > >>>> > >>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote: > >>>>> Hi Ilya, > >>>>> > >>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod > >>>>> Virtual router to be deleted automatically because no VM is running > on > >>>> this > >>>>> vlan but it wasn't deleted. > >>>>> > >>>>> Am I right to expect the VR to be deleted automatically if no VM is > >>>> running > >>>>> in that vlan? > >>>>> > >>>>> Thanks, > >>>>> Nezar. > >>>>> > >>>>> On 15 April 2016 at 09:53, Nezar Madbouh < > nezar.madb...@ipscape.com.au> > >>>>> wrote: > >>>>> > >>>>>> Hi Ilya, > >>>>>> > >>>>>> I have run the script and it didn't return any results: > >>>>>> > >>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh > >>>>>> [root@eqx-cs-mgt-01 ~]# > >>>>>> > >>>>>> > >>>>>> I also ran the command inside mysql: > >>>>>> > >>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id, > >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM > >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like > >>>>>> "pseudoJobDispatcher"; > >>>>>> Empty set (0.00 sec) > >>>>>> > >>>>>> your help is much appreciated, > >>>>>> Nezar. > >>>>>> > >>>>>> > >>>>>> On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> > wrote: > >>>>>> > >>>>>>> It appears you have pending jobs that are not letting you go > through. > >>>>>>> > >>>>>>> Usually stop and starting management service should be enough. > >>>>>>> > >>>>>>> However, try running this script from your management server to > show > >>>> us > >>>>>>> what you have running in async_job table. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> #!/bin/bash > >>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S) > >>>>>>>> DBPASS=$(java -classpath > >>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar > >>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat > >>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password | > >>>> awk > >>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat > >>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1) > >>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep > >>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 ) > >>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep > >>>>>>> db.cloud.username | awk -F'=' '{print $2}') > >>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep > >>>> db.cloud.name > >>>>>>> | awk -F'=' '{print $2}') > >>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep > >>>>>>> db.cloud.port | awk -F'=' '{print $2}') > >>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB" > >>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id, > >>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM > >>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like > >>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote: > >>>>>>>> Hi Simon, > >>>>>>>> > >>>>>>>> The following work was done but unfortunately the problem is not > >>>> fixed: > >>>>>>>> > >>>>>>>> > >>>>>>>> on management server: > >>>>>>>> > >>>>>>>> service cloudstack-management stop > >>>>>>>> > >>>>>>>> > >>>>>>>> on the host: > >>>>>>>> > >>>>>>>> service cloudstack-agent stop > >>>>>>>> > >>>>>>>> service libvirtd restart > >>>>>>>> > >>>>>>>> service cloudstack-agent start > >>>>>>>> > >>>>>>>> > >>>>>>>> on management server: > >>>>>>>> > >>>>>>>> service cloudstack-management start > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Nezar. > >>>>>>>> > >>>>>>>> On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote: > >>>>>>>> > >>>>>>>>> Yes, I think that order is fine. Make sure the processes are > indeed > >>>>>>>>> shutdown before moving on to the next step. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ________________________________________ > >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM > >>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to > >>>> start > >>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>> > >>>>>>>>> Hi Simon, > >>>>>>>>> > >>>>>>>>> Can you please check if the following sequence of action is > correct: > >>>>>>>>> > >>>>>>>>> on management server: > >>>>>>>>> > >>>>>>>>> service cloudstack-management stop > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> on the host: > >>>>>>>>> > >>>>>>>>> service cloudstack-agent stop > >>>>>>>>> > >>>>>>>>> service libvirtd restart > >>>>>>>>> > >>>>>>>>> service cloudstack-agent start > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> on management server: > >>>>>>>>> > >>>>>>>>> service cloudstack-management start > >>>>>>>>> > >>>>>>>>> Thanks very much for your help. > >>>>>>>>> Nezar. > >>>>>>>>> > >>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote: > >>>>>>>>> > >>>>>>>>>> Shutting down all the management services should clear the jobs. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> ________________________________________ > >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM > >>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable > to > >>>>>>> start > >>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>> > >>>>>>>>>> Hi Simon, > >>>>>>>>>> > >>>>>>>>>> I will plan to restart agents, libvirtd and management services. > >>>>>>>>>> > >>>>>>>>>> How do I clear hung jobs. > >>>>>>>>>> > >>>>>>>>>> Your help is much appreciated, > >>>>>>>>>> Nezar. > >>>>>>>>>> > >>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> > wrote: > >>>>>>>>>> > >>>>>>>>>>> Nezar, > >>>>>>>>>>> > >>>>>>>>>>> I've looked through the code related to your release and it > seems > >>>>>>> that > >>>>>>>>> in > >>>>>>>>>>> order to throw this exception, 10 attempts have been made to > >>>> retrieve > >>>>>>>>> the > >>>>>>>>>>> disk information prior to the exception being thrown. > >>>>>>>>>>> > >>>>>>>>>>> It appears there is something really odd going on here. I'm > >>>> starting > >>>>>>>>> to > >>>>>>>>>>> wonder whether it might be worth while for you to stop all of > your > >>>>>>>>>> agents, > >>>>>>>>>>> plus management server processes, restart libvirtd on each of > the > >>>>>>> hosts > >>>>>>>>>> and > >>>>>>>>>>> then bring the agents back up prior to starting the management > >>>>>>> servers. > >>>>>>>>>>> > >>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to > >>>>>>>>> indicate > >>>>>>>>>>> you have some jobs hung, for example: > >>>>>>>>>>> > >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] > >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for > >>>> 1660851 > >>>>>>>>>> seconds > >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] > >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for > >>>> 4253003 > >>>>>>>>>> seconds > >>>>>>>>>>> > >>>>>>>>>>> Restarting the agents, libvirtd and management will not cause > an > >>>>>>>>> outage > >>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, > you > >>>>>>>>> should > >>>>>>>>>>> complete it in a maintenance window. > >>>>>>>>>>> > >>>>>>>>>>> - Si > >>>>>>>>>>> > >>>>>>>>>>> ________________________________________ > >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM > >>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable > to > >>>>>>> start > >>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>> > >>>>>>>>>>> Hi Simon, > >>>>>>>>>>> > >>>>>>>>>>> Please see answers to your questions below. I am not sure if I > >>>>>>> provided > >>>>>>>>>> the > >>>>>>>>>>> correct answer for the primary and secondary storage > >>>> configuration. > >>>>>>>>>> Please > >>>>>>>>>>> let me know if this is not what you are after: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage > >>>>>>>>>> configuration > >>>>>>>>>>> and your primary storage configuration from the UI with the > >>>> details? > >>>>>>>>>>> > >>>>>>>>>>> mysql> select * from configuration where name like > '%secondary%'\G > >>>>>>>>>>> *************************** 1. row *************************** > >>>>>>>>>>> category: Account Defaults > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: max.account.secondary.storage > >>>>>>>>>>> value: 400 > >>>>>>>>>>> description: The default maximum secondary storage space (in > >>>> GiB) > >>>>>>>>> that > >>>>>>>>>>> can be used for an account > >>>>>>>>>>> default_value: 400 > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 2. row *************************** > >>>>>>>>>>> category: Project Defaults > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: max.project.secondary.storage > >>>>>>>>>>> value: 400 > >>>>>>>>>>> description: The default maximum secondary storage space (in > >>>> GiB) > >>>>>>>>> that > >>>>>>>>>>> can be used for a project > >>>>>>>>>>> default_value: 400 > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 3. row *************************** > >>>>>>>>>>> category: Hidden > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: secondary.storage.vm > >>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw== > >>>>>>>>>>> description: Deploys a VM per zone to manage secondary > storage > >>>> if > >>>>>>>>> true, > >>>>>>>>>>> otherwise secondary storage is mounted on management server > >>>>>>>>>>> default_value: NULL > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 4. row *************************** > >>>>>>>>>>> category: Advanced > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: ExtensionRegistry > >>>>>>>>>>> name: secondary.storage.vm.allocators.exclude > >>>>>>>>>>> value: NULL > >>>>>>>>>>> description: Extensions to exclude from being registered > >>>>>>>>>>> default_value: NULL > >>>>>>>>>>> updated: 2014-12-04 00:18:15 > >>>>>>>>>>> scope: Global > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 5. row *************************** > >>>>>>>>>>> category: NetworkManager > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses > >>>>>>>>>>> value: NULL > >>>>>>>>>>> description: Specify the number of secondary ip addresses per > >>>> nic > >>>>>>> per > >>>>>>>>>> vm > >>>>>>>>>>> default_value: 256 > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> 5 rows in set (0.00 sec) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> mysql> select * from configuration where name like > '%primary%'\G > >>>>>>>>>>> *************************** 1. row *************************** > >>>>>>>>>>> category: Account Defaults > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: max.account.primary.storage > >>>>>>>>>>> value: 200 > >>>>>>>>>>> description: The default maximum primary storage space (in > GiB) > >>>>>>> that > >>>>>>>>>> can > >>>>>>>>>>> be used for an account > >>>>>>>>>>> default_value: 200 > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 2. row *************************** > >>>>>>>>>>> category: Project Defaults > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: management-server > >>>>>>>>>>> name: max.project.primary.storage > >>>>>>>>>>> value: 200 > >>>>>>>>>>> description: The default maximum primary storage space (in > GiB) > >>>>>>> that > >>>>>>>>>> can > >>>>>>>>>>> be used for a project > >>>>>>>>>>> default_value: 200 > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> *************************** 3. row *************************** > >>>>>>>>>>> category: Storage > >>>>>>>>>>> instance: DEFAULT > >>>>>>>>>>> component: TemplateManager > >>>>>>>>>>> name: primary.storage.download.wait > >>>>>>>>>>> value: 10800 > >>>>>>>>>>> description: In second, timeout for download template to > primary > >>>>>>>>>> storage > >>>>>>>>>>> default_value: NULL > >>>>>>>>>>> updated: NULL > >>>>>>>>>>> scope: NULL > >>>>>>>>>>> is_dynamic: 0 > >>>>>>>>>>> 3 rows in set (0.00 sec) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Also, what is the Cloudstack version you are running and what > is > >>>> the > >>>>>>> OS > >>>>>>>>>>> you're running it on? > >>>>>>>>>>> > >>>>>>>>>>> Cloudstack 4.4.1 > >>>>>>>>>>> > >>>>>>>>>>> Centos 6.6 > >>>>>>>>>>> > >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line? > >>>>>>>>>>> > >>>>>>>>>>> Primary storage on Hypervisors has the following parameters > >>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on > >>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs > >>>>>>>>>>> > >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Secondary storage on SSVM has the following parameters > >>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on > >>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans= > >>>>>>> 2147483647 > >>>>>>> > >>>> > ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Thanks & Regards, > >>>>>>>>>>> Nezar. > >>>>>>>>>>> > >>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> > wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Also, what is the Cloudstack version you are running and what > is > >>>> the > >>>>>>>>> OS > >>>>>>>>>>>> you're running it on? > >>>>>>>>>>>> > >>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line? > >>>>>>>>>>>> > >>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>> From: Simon Weller <swel...@ena.com> > >>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM > >>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > Unable to > >>>>>>>>> start > >>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>> > >>>>>>>>>>>> Nexar, > >>>>>>>>>>>> > >>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on > >>>> the > >>>>>>>>> host > >>>>>>>>>>>> where the VM resides. > >>>>>>>>>>>> > >>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage > >>>>>>>>>>> configuration > >>>>>>>>>>>> and your primary storage configuration from the UI with the > >>>> details? > >>>>>>>>>>>> > >>>>>>>>>>>> - Si > >>>>>>>>>>>> > >>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM > >>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > Unable to > >>>>>>>>> start > >>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>> > >>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the > SSVM: > >>>>>>>>>>>> > >>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p > 3922 > >>>>>>>>>>>> root@169.254.3.161 > >>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> I can only ssh from one Host: > >>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p > 3922 > >>>>>>>>>>>> root@169.254.3.161 > >>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks & Regards, > >>>>>>>>>>>> Nezar. > >>>>>>>>>>>> > >>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh < > >>>>>>>>> nezar.madb...@ipscape.com.au > >>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was > able > >>>> to > >>>>>>>>>>> create > >>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod > >>>> environment. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Nezar. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> > >>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, > correct? > >>>> So > >>>>>>>>>> you > >>>>>>>>>>>> can > >>>>>>>>>>>>>> write to your primary storage. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM > >>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > Unable > >>>> to > >>>>>>>>>>> start > >>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Simon and Shweta, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Please see the result below: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> First DNS server is 8.8.8.8 > >>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes > >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms > >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms > >>>>>>>>>>>>>> --- 8.8.8.8 ping statistics --- > >>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss > >>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms > >>>>>>>>>>>>>> Good: Can ping DNS server > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> Good: DNS resolves download.cloud.com > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> nfs is currently mounted > >>>>>>>>>>>>>> Mount point is > >>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 > >>>>>>>>>>>>>> Good: Can write to mount point > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity. > >>>>>>>>>>>>>> Good: Can connect to management server port 8250 > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> Good: Java process is running > >>>>>>>>>>>>>> ================================================ > >>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above. > >>>>>>>>>>>>>> root@s-11-VM:~# > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks & Regards, > >>>>>>>>>>>>>> Nezar. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> > >>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Nezar, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Log into the Cloudstack UI. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View > All > >>>>>>>>>> below > >>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" > and > >>>>>>>>>> then > >>>>>>>>>>>>>> click > >>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM). > >>>>>>>>> You'll > >>>>>>>>>>> see > >>>>>>>>>>>> a > >>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and > also > >>>>>>>>> the > >>>>>>>>>>>> Link > >>>>>>>>>>>>>>> Local IP Address line and then follow the original > >>>> instructions > >>>>>>>>> in > >>>>>>>>>>> my > >>>>>>>>>>>>>>> previous email. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - Si > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com> > >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM > >>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > Unable > >>>>>>>>> to > >>>>>>>>>>>> start > >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Nezar, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI . > >>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the > >>>> host > >>>>>>>>>>>> details > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>> system vms. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>> Shweta > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM > >>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > Unable > >>>>>>>>> to > >>>>>>>>>>>> start > >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> How can I find out which host the SSVM is on? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks & Regards, > >>>>>>>>>>>>>>> Nezar. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from > >>>> the > >>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the > host > >>>>>>>>> the > >>>>>>>>>>> SSVM > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>> and issuing this command as root: ssh -i > >>>>>>>>> /root/.ssh/id_rsa.cloud > >>>>>>>>>>> -p > >>>>>>>>>>>>>> 3922 > >>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link > >>>>>>>>> local > >>>>>>>>>>> ip). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Try running the SSVM diangnostic script: > >>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - Si > >>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM > >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > >>>> Unable > >>>>>>>>>> to > >>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Please see details below: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host > >>>>>>>>> where > >>>>>>>>>>> id = > >>>>>>>>>>>>>> 5; > >>>>>>>>>>>>>>>> Reading table information for completion of table and > column > >>>>>>>>>> names > >>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup > with > >>>> -A > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Database changed > >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ > >>>>>>>>>>>>>>>> | id | name | status | type | > >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ > >>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | > >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ > >>>>>>>>>>>>>>>> 1 row in set (0.00 sec) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log > >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: > >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't > find > >>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) > >>>>>>>>>>>>>>>> at > >>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>> > >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) > >>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >>>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list > >>>>>>>>>>>>>>>> Name State Autostart > >>>>>>>>>>>>>>>> ----------------------------------------- > >>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no > >>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no > >>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h > >>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on > >>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root > >>>>>>>>>>>>>>>> 203G 215G 1.3G 1% / > >>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm > >>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot > >>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary > >>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54% > >>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain > with > >>>>>>>>>>>> matching > >>>>>>>>>>>>>>> name > >>>>>>>>>>>>>>>> 'i-11-600-VM' > >>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: > no > >>>>>>>>>>> storage > >>>>>>>>>>>>>> vol > >>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Your help is much appreciated, > >>>>>>>>>>>>>>>> Nezar > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Nezar, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume > >>>>>>>>>> prior > >>>>>>>>>>> to > >>>>>>>>>>>>>>>>> throwing the host exception. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into > your > >>>>>>>>>>> MySQL > >>>>>>>>>>>>>>> server > >>>>>>>>>>>>>>>>> and run this: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where > id=5; > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Can you post the results? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> If it's a host, can you post the > >>>>>>>>>>>> /var/log/cloudstack/agent/agent.log > >>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>> that host? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> - Si > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM > >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > >>>>>>>>> Unable > >>>>>>>>>>> to > >>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Simon, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> When you say "double check your routers tied to the > >>>>>>>>> network", > >>>>>>>>>> do > >>>>>>>>>>>> you > >>>>>>>>>>>>>>> mean > >>>>>>>>>>>>>>>>> the physical routers or the virtual routers? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I > >>>>>>>>>> can't > >>>>>>>>>>>> ping > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> Prod virtual router from management server or Host. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server > and > >>>>>>>>>>> Host > >>>>>>>>>>>>>> and I > >>>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>>> able to create a VM in Dev Environment. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I > launched > >>>>>>>>> the > >>>>>>>>>>>>>> console > >>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>> couldn't login. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Please note that we need to take into consideration that > the > >>>>>>>>>>>>>> affected > >>>>>>>>>>>>>>>>> network is the Production network and we need to minimise > >>>>>>>>>>> service > >>>>>>>>>>>>>>> outage > >>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>> much as possible. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Your help is much appreciated, > >>>>>>>>>>>>>>>>> Nezar > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com > > > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed > >>>>>>>>> and > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> management server can't communicate with it. > >>>>>>>>>>>>>>>>>> Are you using an advanced network setup? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network > where > >>>>>>>>>> you > >>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>> attempting to start an instance. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> - Si > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM > >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > >>>>>>>>>> Unable > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi Glenn, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to > >>>>>>>>>>> create > >>>>>>>>>>>> a > >>>>>>>>>>>>>> VM > >>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>> gives me the same error message but on different host. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service > >>>>>>>>>>> affecting? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks very much for your help. > >>>>>>>>>>>>>>>>>> Neza > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner < > >>>>>>>>>>>>>> glenn.wag...@shapeblue.com> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts > >>>>>>>>> , > >>>>>>>>>>> what > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor > >>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is > >>>>>>>>>>>>>> reporting, > >>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>> Glenn > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com> > >>>>>>>>>>>>>>>>>>> Glenn Wagner > >>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue > >>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091* > >>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> | > >>>>>>>>>>>>>>>> m: > >>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111> > >>>>>>>>>>>>>>>>>>> e: *glenn.wag...@shapeblue.com | t: * > >>>>>>>>>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * > >>>>>>>>>>>>>> www.shapeblue.com > >>>>>>>>>>>>>>> * > >>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com> > >>>>>>>>>>>>>>>>>>> a: > >>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West > >>>>>>>>>> Cape > >>>>>>>>>>>> Town > >>>>>>>>>>>>>>> 7130 > >>>>>>>>>>>>>>>>>> South Africa > >>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England & > >>>>>>>>>> Wales. > >>>>>>>>>>>>>>>> ShapeBlue > >>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India > >>>>>>>>> and > >>>>>>>>>> is > >>>>>>>>>>>>>>> operated > >>>>>>>>>>>>>>>>>> under > >>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil > >>>>>>>>> Consultoria > >>>>>>>>>>>> Ltda > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under > >>>>>>>>>> license > >>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> Shape > >>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered > >>>>>>>>> by > >>>>>>>>>>> The > >>>>>>>>>>>>>>>> Republic > >>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape > Blue > >>>>>>>>>>> Ltd. > >>>>>>>>>>>>>>>> ShapeBlue > >>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>> a registered trademark. > >>>>>>>>>>>>>>>>>>> This email and any attachments to it may be > confidential > >>>>>>>>>> and > >>>>>>>>>>>> are > >>>>>>>>>>>>>>>>> intended > >>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is > >>>>>>>>>>> addressed. > >>>>>>>>>>>>>> Any > >>>>>>>>>>>>>>>> views > >>>>>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and > do > >>>>>>>>>> not > >>>>>>>>>>>>>>>>> necessarily > >>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies. > >>>>>>>>> If > >>>>>>>>>>> you > >>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take > >>>>>>>>>> any > >>>>>>>>>>>>>> action > >>>>>>>>>>>>>>>>> based > >>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. > Please > >>>>>>>>>>>> contact > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> sender > >>>>>>>>>>>>>>>>>>> if you believe you have received this email in error. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto: > >>>>>>>>> nezar.madb...@ipscape.com.au] > >>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM > >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: > >>>>>>>>>>> Unable > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi Jeremy, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I have uploaded the contents to > >>>>>>>>>>> http://pastebin.com/1J3mmgTh > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> your help is much appreciated, > >>>>>>>>>>>>>>>>>>> Nezar. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson < > >>>>>>>>>>>>>> jpeter...@acentek.net> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin. > >>>>>>>>>>>> That's > >>>>>>>>>>>>>>>> easier > >>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed > >>>>>>>>>> before > >>>>>>>>>>>>>> being > >>>>>>>>>>>>>>>> sent > >>>>>>>>>>>>>>>>>>>> to user group. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Jeremy > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -------- Original message -------- > >>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > >>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00) > >>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org > >>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message: > >>>>>>>>>> Unable > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> start > >>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class > >>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which > >>>>>>>>>> contains > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> logs > >>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600). > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Your help is much appreciated. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>> Nezar. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of > >>>>>>>>> CloudStack > >>>>>>>>>>>>>> related > >>>>>>>>>>>>>>>>>> services: > >>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build > >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | > >>>>>>>>>>>> CSForge > >>>>>>>>>>>>>> - > >>>>>>>>>>>>>>>> rapid > >>>>>>>>>>>>>>>>>>> IaaS deployment framework < > >>>>>>>>> http://shapeblue.com/csforge/> > >>>>>>>>>>>>>>>>>>> CloudStack Consulting < > >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/> > >>>>>>>>>>>>>>>> | > >>>>>>>>>>>>>>>>>> CloudStack > >>>>>>>>>>>>>>>>>>> Software Engineering > >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/ > > > >>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support > >>>>>>>>>>>>>>>>>>> < > >>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/> > >>>>>>>>>> | > >>>>>>>>>>>>>>>> CloudStack > >>>>>>>>>>>>>>>>>>> Bootcamp Training Courses < > >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/ > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> DISCLAIMER > >>>>>>>>>>>>>>> ========== > >>>>>>>>>>>>>>> This e-mail may contain privileged and confidential > >>>> information > >>>>>>>>>>> which > >>>>>>>>>>>> is > >>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. > It > >>>> is > >>>>>>>>>>>>>> intended > >>>>>>>>>>>>>>> only for the use of the individual or entity to which it is > >>>>>>>>>>> addressed. > >>>>>>>>>>>>>> If > >>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized > to > >>>>>>>>>> read, > >>>>>>>>>>>>>> retain, > >>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have > >>>>>>>>> received > >>>>>>>>>>> this > >>>>>>>>>>>>>>> communication in error, please notify the sender and delete > >>>> all > >>>>>>>>>>> copies > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business > does > >>>> not > >>>>>>>>>>>> accept > >>>>>>>>>>>>>> any > >>>>>>>>>>>>>>> liability for virus infected mails. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >
