Looks like I'm getting the same issue after an upgrade, this is with all my System VMs. Normal VMs are not starting either.
2014-09-27 15:24:02,437 WARN [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-df72ecab) Exception while trying to start secondary > storage vm > java.lang.RuntimeException: Job failed due to exception Resource > [Host:101] is unreachable: Host 101: Unable to start instance due to Unable > to get answer that is of class com.cloud.agent.api.StartAnswer > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2014-09-27 15:24:02,438 INFO [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-df72ecab) Unable to start secondary storage vm for > standby capacity, secStorageVm vm Id : 312, will recycle it and start a new > one On Wed, Aug 27, 2014 at 10:31 AM, Ugo Vasi <ugo.v...@procne.it> wrote: > Hi, > all the “cache_mode" value were "none", in lowercase: > > mysql> select id,name,cache_mode from disk_offering; > +----+------------------------------------------+------------+ > | id | name | cache_mode | > +----+------------------------------------------+------------+ > | 1 | Small Instance | none | > | 2 | Medium Instance | none | > | 3 | Small | none | > | 4 | Medium | none | > | 5 | Large | none | > | 6 | Custom | none | > | 7 | System Offering For Internal LB VM | none | > | 8 | System Offering For Software Router | none | > | 9 | System Offering For Secondary Storage VM | none | > | 10 | System Offering For Console Proxy | none | > | 11 | System Offering For Elastic LB VM | none | > +----+------------------------------------------+------------+ > > I tried to set all uppercase: > mysql> update disk_offering set cache_mode='NONE'; > Query OK, 11 rows affected (0.05 sec) > Rows matched: 11 Changed: 11 Warnings: 0 > > mysql> select id,name,cache_mode from disk_offering; > +----+------------------------------------------+------------+ > | id | name | cache_mode | > +----+------------------------------------------+------------+ > | 1 | Small Instance | NONE | > | 2 | Medium Instance | NONE | > | 3 | Small | NONE | > | 4 | Medium | NONE | > | 5 | Large | NONE | > | 6 | Custom | NONE | > | 7 | System Offering For Internal LB VM | NONE | > | 8 | System Offering For Software Router | NONE | > | 9 | System Offering For Secondary Storage VM | NONE | > | 10 | System Offering For Console Proxy | NONE | > | 11 | System Offering For Elastic LB VM | NONE | > +----+------------------------------------------+------------+ > > But the router is not starting. The message on web interface is: > > Job failed due to exception Resource [Host:4] is unreachable: Host 4: > Unable to start instance due to Unable to get answer that is of class > com.cloud.agent.api.StartAnswer > > I set the cache_mode values in the database as before, in lowercase. > > > > Il 27/08/2014 13:37, Harikrishna Patnala ha scritto: > > Hi, >> >> Can you check the value of “cache_mode" column in disk_offering table for >> the disk_offering used by router. >> As per the fix “cache_mode" value should be in upper case. >> >> -Harikrishna >> >> On 27-Aug-2014, at 3:00 pm, Erik Weber <terbol...@gmail.com> wrote: >> >> Most likely this: https://issues.apache.org/jira/browse/CLOUDSTACK-6893 >>> >>> I'm not aware of a manual workaround, perhaps someone else is. >>> >>> -- >>> Erik >>> >>> >>> On Wed, Aug 27, 2014 at 11:12 AM, Ugo Vasi <ugo.v...@procne.it> wrote: >>> >>> Hi all, >>>> we are trying to upgrade a cloustack 4.3 installation to a 4.4 version. >>>> Manager and agents (and ssvm and cpvm) runs in a new version and seems >>>> to >>>> be OK but virtual routers are still 4.3.0. >>>> >>>> I tried to reboot a router but this does not restart it gives me these >>>> error messages: >>>> >>>> ... >>>> 2014-08-27 10:50:01,328 DEBUG [c.c.a.t.Request] >>>> (AgentManager-Handler-5:null) Seq 1-7961519716260912076: Processing: { >>>> Ans: , MgmtId: 240206110326297, via: 1, Ver: v1, Flags: 10, >>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"No enum >>>> constant com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef. >>>> diskCacheMode.none","wait":0}},{"com.cloud.agent.api.Answer" >>>> :{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{" >>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{" >>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{" >>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{" >>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}},{"com.cloud.agent.api.Answer":{" >>>> result":false,"details":"Stopped by previous failure","wait":0}},{"com. >>>> cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous >>>> failure","wait":0}}] } >>>> 2014-08-27 10:50:01,328 DEBUG [c.c.a.t.Request] >>>> (Work-Job-Executor-13:ctx-3a10ee96 >>>> job-120/job-121 ctx-2cd18d35) Seq 1-7961519716260912076: Received: { >>>> Ans: , >>>> MgmtId: 240206110326297, via: 1, Ver: v1, Flags: 10, { Answer, Answer, >>>> Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, >>>> Answer, Answer } } >>>> 2014-08-27 10:50:01,367 ERROR [c.c.v.VirtualMachineManagerImpl] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35) Failed >>>> to start instance VM[DomainRouter|r-12-VM] >>>> com.cloud.utils.exception.CloudRuntimeException: Unable to get answer >>>> that is of class com.cloud.agent.api.StartAnswer >>>> at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) >>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >>>> VirtualMachineManagerImpl.java:1019) >>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >>>> VirtualMachineManagerImpl.java:5190) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ... >>>> >>>> 2014-08-27 10:50:01,829 DEBUG [c.c.c.CapacityManagerImpl] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35) >>>> release >>>> mem from host: 1, old used: 2952790016,reserved: 0, total: 50640633856; >>>> new >>>> used: 2818572288,reserved:0; movedfromreserved: >>>> false,moveToReserveredfalse >>>> 2014-08-27 10:50:01,865 ERROR [c.c.v.VmWorkJobHandlerProxy] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35) >>>> Invocation exception, caused by: com.cloud.exception. >>>> AgentUnavailableException: >>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance due >>>> to >>>> Unable to get answer that is of class com.cloud.agent.api.StartAnswer >>>> 2014-08-27 10:50:01,866 INFO [c.c.v.VmWorkJobHandlerProxy] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121 ctx-2cd18d35) >>>> Rethrow >>>> exception com.cloud.exception.AgentUnavailableException: Resource >>>> [Host:1] is unreachable: Host 1: Unable to start instance due to Unable >>>> to >>>> get answer that is of class com.cloud.agent.api.StartAnswer >>>> 2014-08-27 10:50:01,866 DEBUG [c.c.v.VmWorkJobDispatcher] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121) Done with run of VM >>>> work job: com.cloud.vm.VmWorkStart for VM 12, job origin: 120 >>>> 2014-08-27 10:50:01,867 ERROR [c.c.v.VmWorkJobDispatcher] >>>> (Work-Job-Executor-13:ctx-3a10ee96 job-120/job-121) Unable to complete >>>> AsyncJobVO {id:121, userId: 2, accountId: 2, instanceType: null, >>>> instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: >>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oA >>>> BGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50 >>>> UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5n >>>> L0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYX >>>> ZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB- >>>> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS >>>> 91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV2 >>>> 9ya5- >>>> ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmR >>>> sZXJOYW1lcQB- >>>> AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAADHQAGVZpcnR1YWxNYWNoaW5l >>>> TWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNo >>>> TWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_ >>>> QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55 >>>> QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZk >>>> bUZzZFdWNGNBRXhw, >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >>>> result: null, initMsid: 240206110326297, completeMsid: null, >>>> lastUpdated: >>>> null, lastPolled: null, created: Wed Aug 27 10:49:59 CEST 2014}, job >>>> origin:120 >>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is >>>> unreachable: Host 1: Unable to start instance due to Unable to get >>>> answer >>>> that is of class com.cloud.agent.api.StartAnswer >>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >>>> VirtualMachineManagerImpl.java:1105) >>>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >>>> VirtualMachineManagerImpl.java:5190) >>>> .... >>>> >>>> I tried to destroy the router but the new one fails to start in the same >>>> way. >>>> >>>> The other router (4.3.0 version) is still working but I have not tried >>>> to >>>> restart it. >>>> >>>> Of course the vmon the network with the router ko can not get out of the >>>> network. >>>> >>>> Management run on ubuntu 12.04, Agents run on ubuntu 14.04. In another >>>> similar installation of cloudstack 4.3 it's all quite ok (I found a >>>> problem >>>> with a guest network, see 'Unable to start a VM due to insufficient >>>> capacity' precedent mailing list messages). >>>> >>>> -- >>>> >>>> U g o V a s i <ugo.v...@procne.it> >>>> P r o c n e s.r.l >) >>>> via Cotonificio 45 33010 Tavagnacco IT >>>> phone: +390432486523 fax: +390432486523 >>>> >>>> Le informazioni contenute in questo messaggio sono riservate e >>>> confidenziali ed è vietata la diffusione in qualunque modo eseguita. >>>> Qualora Lei non fosse la persona a cui il presente messaggio è >>>> destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene >>>> gentilmente comunicazione. >>>> Per qualsiasi informazione si prega di contattare supp...@procne.it . >>>> Rif. D.L. 196/2003 >>>> >>>> >>>> >> >> >> >> > > -- > > U g o V a s i <ugo.v...@procne.it> > P r o c n e s.r.l >) > via Cotonificio 45 33010 Tavagnacco IT > phone: +390432486523 fax: +390432486523 > > Le informazioni contenute in questo messaggio sono riservate e > confidenziali ed è vietata la diffusione in qualunque modo eseguita. > Qualora Lei non fosse la persona a cui il presente messaggio è > destinato, La invitiamo ad eliminarlo e a non leggerlo, dandocene > gentilmente comunicazione. > Per qualsiasi informazione si prega di contattare supp...@procne.it . > Rif. D.L. 196/2003 > >
