Hi Ilya,
I have run the script and it didn't return any results:
[root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
[root@eqx-cs-mgt-01 ~]#
I also ran the command inside mysql:
mysql> SELECT id, user_id, account_id, instance_type, instance_id, job_cmd,
job_cmd_info, job_dispatcher, job_pending_signals FROM cloud.async_job
where job_status=0 and job_dispatcher not like "pseudoJobDispatcher";
Empty set (0.00 sec)
your help is much appreciated,
Nezar.
On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> wrote:
> It appears you have pending jobs that are not letting you go through.
>
> Usually stop and starting management service should be enough.
>
> However, try running this script from your management server to show us
> what you have running in async_job table.
>
>
>
> > #!/bin/bash
> > DATESTAMP=$(date +%m%d%y-%H%M%S)
> > DBPASS=$(java -classpath
> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.username | awk -F'=' '{print $2}')
> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
> | awk -F'=' '{print $2}')
> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.port | awk -F'=' '{print $2}')
> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> > echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> cloud.async_job where job_status=0 and job_dispatcher not like
> "pseudoJobDispatcher"' | $MYSQLCMD
>
>
>
>
> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> > Hi Simon,
> >
> > The following work was done but unfortunately the problem is not fixed:
> >
> >
> > on management server:
> >
> > service cloudstack-management stop
> >
> >
> > on the host:
> >
> > service cloudstack-agent stop
> >
> > service libvirtd restart
> >
> > service cloudstack-agent start
> >
> >
> > on management server:
> >
> > service cloudstack-management start
> >
> > Regards,
> > Nezar.
> >
> > On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote:
> >
> >> Yes, I think that order is fine. Make sure the processes are indeed
> >> shutdown before moving on to the next step.
> >>
> >>
> >>
> >>
> >>
> >> ________________________________________
> >> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >> Sent: Tuesday, March 22, 2016 5:33 PM
> >> To: users@cloudstack.apache.org
> >> Subject: Re: Unable to create a New VM - Error message: Unable to start
> >> instance due to Unable to get answer that is of class
> >> com.cloud.agent.api.StartAnswer
> >>
> >> Hi Simon,
> >>
> >> Can you please check if the following sequence of action is correct:
> >>
> >> on management server:
> >>
> >> service cloudstack-management stop
> >>
> >>
> >> on the host:
> >>
> >> service cloudstack-agent stop
> >>
> >> service libvirtd restart
> >>
> >> service cloudstack-agent start
> >>
> >>
> >> on management server:
> >>
> >> service cloudstack-management start
> >>
> >> Thanks very much for your help.
> >> Nezar.
> >>
> >> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote:
> >>
> >>> Shutting down all the management services should clear the jobs.
> >>>
> >>>
> >>> ________________________________________
> >>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Unable to create a New VM - Error message: Unable to start
> >>> instance due to Unable to get answer that is of class
> >>> com.cloud.agent.api.StartAnswer
> >>>
> >>> Hi Simon,
> >>>
> >>> I will plan to restart agents, libvirtd and management services.
> >>>
> >>> How do I clear hung jobs.
> >>>
> >>> Your help is much appreciated,
> >>> Nezar.
> >>>
> >>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> wrote:
> >>>
> >>>> Nezar,
> >>>>
> >>>> I've looked through the code related to your release and it seems that
> >> in
> >>>> order to throw this exception, 10 attempts have been made to retrieve
> >> the
> >>>> disk information prior to the exception being thrown.
> >>>>
> >>>> It appears there is something really odd going on here. I'm starting
> >> to
> >>>> wonder whether it might be worth while for you to stop all of your
> >>> agents,
> >>>> plus management server processes, restart libvirtd on each of the
> hosts
> >>> and
> >>>> then bring the agents back up prior to starting the management
> servers.
> >>>>
> >>>> There are a bunch of logs the pastebin you posted that seems to
> >> indicate
> >>>> you have some jobs hung, for example:
> >>>>
> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
> >>> seconds
> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
> >>> seconds
> >>>>
> >>>> Restarting the agents, libvirtd and management will not cause an
> >> outage
> >>>> to your cloud, but I would suggest, if you decided to do this, you
> >> should
> >>>> complete it in a maintenance window.
> >>>>
> >>>> - Si
> >>>>
> >>>> ________________________________________
> >>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>> To: users@cloudstack.apache.org
> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> start
> >>>> instance due to Unable to get answer that is of class
> >>>> com.cloud.agent.api.StartAnswer
> >>>>
> >>>> Hi Simon,
> >>>>
> >>>> Please see answers to your questions below. I am not sure if I
> provided
> >>> the
> >>>> correct answer for the primary and secondary storage configuration.
> >>> Please
> >>>> let me know if this is not what you are after:
> >>>>
> >>>>
> >>>> Can you screenshot and post somewhere your secondary storage
> >>> configuration
> >>>> and your primary storage configuration from the UI with the details?
> >>>>
> >>>> mysql> select * from configuration where name like '%secondary%'\G
> >>>> *************************** 1. row ***************************
> >>>> category: Account Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.account.secondary.storage
> >>>> value: 400
> >>>> description: The default maximum secondary storage space (in GiB)
> >> that
> >>>> can be used for an account
> >>>> default_value: 400
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 2. row ***************************
> >>>> category: Project Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.project.secondary.storage
> >>>> value: 400
> >>>> description: The default maximum secondary storage space (in GiB)
> >> that
> >>>> can be used for a project
> >>>> default_value: 400
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 3. row ***************************
> >>>> category: Hidden
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: secondary.storage.vm
> >>>> value: +rwRhJOXPymbQOuAMLnhiw==
> >>>> description: Deploys a VM per zone to manage secondary storage if
> >> true,
> >>>> otherwise secondary storage is mounted on management server
> >>>> default_value: NULL
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 4. row ***************************
> >>>> category: Advanced
> >>>> instance: DEFAULT
> >>>> component: ExtensionRegistry
> >>>> name: secondary.storage.vm.allocators.exclude
> >>>> value: NULL
> >>>> description: Extensions to exclude from being registered
> >>>> default_value: NULL
> >>>> updated: 2014-12-04 00:18:15
> >>>> scope: Global
> >>>> is_dynamic: 0
> >>>> *************************** 5. row ***************************
> >>>> category: NetworkManager
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: vm.network.nic.max.secondary.ipaddresses
> >>>> value: NULL
> >>>> description: Specify the number of secondary ip addresses per nic
> per
> >>> vm
> >>>> default_value: 256
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> 5 rows in set (0.00 sec)
> >>>>
> >>>>
> >>>> mysql> select * from configuration where name like '%primary%'\G
> >>>> *************************** 1. row ***************************
> >>>> category: Account Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.account.primary.storage
> >>>> value: 200
> >>>> description: The default maximum primary storage space (in GiB) that
> >>> can
> >>>> be used for an account
> >>>> default_value: 200
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 2. row ***************************
> >>>> category: Project Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.project.primary.storage
> >>>> value: 200
> >>>> description: The default maximum primary storage space (in GiB) that
> >>> can
> >>>> be used for a project
> >>>> default_value: 200
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 3. row ***************************
> >>>> category: Storage
> >>>> instance: DEFAULT
> >>>> component: TemplateManager
> >>>> name: primary.storage.download.wait
> >>>> value: 10800
> >>>> description: In second, timeout for download template to primary
> >>> storage
> >>>> default_value: NULL
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> 3 rows in set (0.00 sec)
> >>>>
> >>>>
> >>>>
> >>>> Also, what is the Cloudstack version you are running and what is the
> OS
> >>>> you're running it on?
> >>>>
> >>>> Cloudstack 4.4.1
> >>>>
> >>>> Centos 6.6
> >>>>
> >>>> Are you using actimeo=0 in your NFS mount line?
> >>>>
> >>>> Primary storage on Hypervisors has the following parameters
> >>>> 10.61.154.10:/Cloudstack/primary on
> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>
> >>>>
> >>>> Secondary storage on SSVM has the following parameters
> >>>> 10.61.154.10:/Cloudstack/secondary on
> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>
> >>>>
> >>>
> >>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>
> >>>>
> >>>> Thanks & Regards,
> >>>> Nezar.
> >>>>
> >>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> wrote:
> >>>>
> >>>>> Also, what is the Cloudstack version you are running and what is the
> >> OS
> >>>>> you're running it on?
> >>>>>
> >>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Simon Weller <swel...@ena.com>
> >>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >> start
> >>>>> instance due to Unable to get answer that is of class
> >>>>> com.cloud.agent.api.StartAnswer
> >>>>>
> >>>>> Nexar,
> >>>>>
> >>>>> That's normal behaviour. You can only ssh to the link local on the
> >> host
> >>>>> where the VM resides.
> >>>>>
> >>>>> Can you screenshot and post somewhere your secondary storage
> >>>> configuration
> >>>>> and your primary storage configuration from the UI with the details?
> >>>>>
> >>>>> - Si
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >> start
> >>>>> instance due to Unable to get answer that is of class
> >>>>> com.cloud.agent.api.StartAnswer
> >>>>>
> >>>>> Hi Simon,
> >>>>>
> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
> >>>>>
> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>> root@169.254.3.161
> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>
> >>>>>
> >>>>> I can only ssh from one Host:
> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>> root@169.254.3.161
> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>
> >>>>> Thanks & Regards,
> >>>>> Nezar.
> >>>>>
> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >> nezar.madb...@ipscape.com.au
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Simon,
> >>>>>>
> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to
> >>>> create
> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Nezar.
> >>>>>>
> >>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote:
> >>>>>>
> >>>>>>> I'm assuming you have other VMs successfully running, correct? So
> >>> you
> >>>>> can
> >>>>>>> write to your primary storage.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>> To: users@cloudstack.apache.org
> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>> start
> >>>>>>> instance due to Unable to get answer that is of class
> >>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>
> >>>>>>> Hi Simon and Shweta,
> >>>>>>>
> >>>>>>> Please see the result below:
> >>>>>>>
> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>> ================================================
> >>>>>>> First DNS server is 8.8.8.8
> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>> Good: Can ping DNS server
> >>>>>>> ================================================
> >>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>> ================================================
> >>>>>>> nfs is currently mounted
> >>>>>>> Mount point is
> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>> Good: Can write to mount point
> >>>>>>> ================================================
> >>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>> Good: Can connect to management server port 8250
> >>>>>>> ================================================
> >>>>>>> Good: Java process is running
> >>>>>>> ================================================
> >>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>> root@s-11-VM:~#
> >>>>>>>
> >>>>>>> Thanks & Regards,
> >>>>>>> Nezar.
> >>>>>>>
> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote:
> >>>>>>>
> >>>>>>>> Nezar,
> >>>>>>>>
> >>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>
> >>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
> >>> below
> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
> >>> then
> >>>>>>> click
> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >> You'll
> >>>> see
> >>>>> a
> >>>>>>>> bunch of information about the VM. Find the Host line and also
> >> the
> >>>>> Link
> >>>>>>>> Local IP Address line and then follow the original instructions
> >> in
> >>>> my
> >>>>>>>> previous email.
> >>>>>>>>
> >>>>>>>> - Si
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com>
> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >> to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Hi Nezar,
> >>>>>>>>
> >>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>> You can also use list api command listSystemVms to get the host
> >>>>> details
> >>>>>>> of
> >>>>>>>> system vms.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Shweta
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >> to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>
> >>>>>>>> Thanks & Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com>
> >> wrote:
> >>>>>>>>
> >>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>
> >>>>>>>>> This can be done by getting the link-local-ip address from the
> >>>>>>>>> infrastructure menu in the UI and then sshing into the host
> >> the
> >>>> SSVM
> >>>>>>> is
> >>>>>>>> on
> >>>>>>>>> and issuing this command as root: ssh -i
> >> /root/.ssh/id_rsa.cloud
> >>>> -p
> >>>>>>> 3922
> >>>>>>>>> root@link-local-ip (replace link-local-ip with the link
> >> local
> >>>> ip).
> >>>>>>>>>
> >>>>>>>>> Try running the SSVM diangnostic script:
> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>
> >>>>>>>>> - Si
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >>> to
> >>>>>>> start
> >>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Please see details below:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >> where
> >>>> id =
> >>>>>>> 5;
> >>>>>>>>> Reading table information for completion of table and column
> >>> names
> >>>>>>>>> You can turn off this feature to get a quicker startup with -A
> >>>>>>>>>
> >>>>>>>>> Database changed
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> | id | name | status | type |
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>> at
> >> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>> at
> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>> Name State Autostart
> >>>>>>>>> -----------------------------------------
> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>> 203G 215G 1.3G 1% /
> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>> 7.6T 4.1T 3.5T 54%
> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>
> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
> >>>>> matching
> >>>>>>>> name
> >>>>>>>>> 'i-11-600-VM'
> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
> >>>> storage
> >>>>>>> vol
> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>
> >>>>>>>>> Your help is much appreciated,
> >>>>>>>>> Nezar
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Nezar,
> >>>>>>>>>>
> >>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>
> >>>>>>>>>> I see it complaining about the inability to find a volume
> >>> prior
> >>>> to
> >>>>>>>>>> throwing the host exception.
> >>>>>>>>>>
> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
> >>>> MySQL
> >>>>>>>> server
> >>>>>>>>>> and run this:
> >>>>>>>>>>
> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
> >>>>>>>>>>
> >>>>>>>>>> Can you post the results?
> >>>>>>>>>>
> >>>>>>>>>> If it's a host, can you post the
> >>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>> from
> >>>>>>>>>> that host?
> >>>>>>>>>>
> >>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>
> >>>>>>>>>> - Si
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >> Unable
> >>>> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon,
> >>>>>>>>>>
> >>>>>>>>>> When you say "double check your routers tied to the
> >> network",
> >>> do
> >>>>> you
> >>>>>>>> mean
> >>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>
> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>> can't
> >>>>> ping
> >>>>>>>> the
> >>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>
> >>>>>>>>>> But I can ping Dev virtual router from management server and
> >>>> Host
> >>>>>>> and I
> >>>>>>>>> was
> >>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>
> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
> >> the
> >>>>>>> console
> >>>>>>>>> but
> >>>>>>>>>> couldn't login.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Please note that we need to take into consideration that the
> >>>>>>> affected
> >>>>>>>>>> network is the Production network and we need to minimise
> >>>> service
> >>>>>>>> outage
> >>>>>>>>> as
> >>>>>>>>>> much as possible.
> >>>>>>>>>>
> >>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>> Nezar
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >> and
> >>>> the
> >>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>
> >>>>>>>>>>> If so, double check your routers tied to the network where
> >>> you
> >>>>> are
> >>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>> Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>
> >>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>
> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>> create
> >>>>> a
> >>>>>>> VM
> >>>>>>>> it
> >>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>
> >>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>> affecting?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>> Neza
> >>>>>>>>>>>
> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>> glenn.wag...@shapeblue.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you explain your environment first , how many hosts
> >> ,
> >>>> what
> >>>>>>> is
> >>>>>>>>> your
> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>
> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>> reporting,
> >>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>> Glenn
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>> Senior Consultant , ShapeBlue
> >>>>>>>>>>>> d: * | s: +27 21 527 0091*
> >>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
> >>>>>>>>> m:
> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>> e: *glenn.wag...@shapeblue.com | t: *
> >>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:> | w: *
> >>>>>>> www.shapeblue.com
> >>>>>>>> *
> >>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>> a:
> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>> Cape
> >>>>> Town
> >>>>>>>> 7130
> >>>>>>>>>>> South Africa
> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>> Wales.
> >>>>>>>>> ShapeBlue
> >>>>>>>>>>>> Services India LLP is a company incorporated in India
> >> and
> >>> is
> >>>>>>>> operated
> >>>>>>>>>>> under
> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >> Consultoria
> >>>>> Ltda
> >>>>>>> is
> >>>>>>>> a
> >>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>> license
> >>>>>>> from
> >>>>>>>>> Shape
> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >> by
> >>>> The
> >>>>>>>>> Republic
> >>>>>>>>>> of
> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue
> >>>> Ltd.
> >>>>>>>>> ShapeBlue
> >>>>>>>>>>> is
> >>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>> This email and any attachments to it may be confidential
> >>> and
> >>>>> are
> >>>>>>>>>> intended
> >>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>> addressed.
> >>>>>>> Any
> >>>>>>>>> views
> >>>>>>>>>>> or
> >>>>>>>>>>>> opinions expressed are solely those of the author and do
> >>> not
> >>>>>>>>>> necessarily
> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >> If
> >>>> you
> >>>>>>> are
> >>>>>>>>> not
> >>>>>>>>>>> the
> >>>>>>>>>>>> intended recipient of this email, you must neither take
> >>> any
> >>>>>>> action
> >>>>>>>>>> based
> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
> >>>>> contact
> >>>>>>>> the
> >>>>>>>>>>> sender
> >>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >> nezar.madb...@ipscape.com.au]
> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>> Unable
> >>>>>>> to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have uploaded the contents to
> >>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>
> >>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>> jpeter...@acentek.net>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>> That's
> >>>>>>>>> easier
> >>>>>>>>>>>>> to look through as I think attachments are removed
> >>> before
> >>>>>>> being
> >>>>>>>>> sent
> >>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>> Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please find attached managerment-server.log which
> >>> contains
> >>>>> the
> >>>>>>>> logs
> >>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >> CloudStack
> >>>>>>> related
> >>>>>>>>>>> services:
> >>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>> CSForge
> >>>>>>> -
> >>>>>>>>> rapid
> >>>>>>>>>>>> IaaS deployment framework <
> >> http://shapeblue.com/csforge/>
> >>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>> |
> >>>>>>>>>>> CloudStack
> >>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
> >>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>> <
> >> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>> |
> >>>>>>>>> CloudStack
> >>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> DISCLAIMER
> >>>>>>>> ==========
> >>>>>>>> This e-mail may contain privileged and confidential information
> >>>> which
> >>>>> is
> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is
> >>>>>>> intended
> >>>>>>>> only for the use of the individual or entity to which it is
> >>>> addressed.
> >>>>>>> If
> >>>>>>>> you are not the intended recipient, you are not authorized to
> >>> read,
> >>>>>>> retain,
> >>>>>>>> copy, print, distribute or use this message. If you have
> >> received
> >>>> this
> >>>>>>>> communication in error, please notify the sender and delete all
> >>>> copies
> >>>>>>> of
> >>>>>>>> this message. Accelerite, a Persistent Systems business does not
> >>>>> accept
> >>>>>>> any
> >>>>>>>> liability for virus infected mails.
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
