Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-21 Thread Wido den Hollander 


On 10/21/2014 12:36 AM, Nux! wrote:
 Wido,
 
 Alas DEBUG is on without telling me anything I can understand. Here's some 
 more:
 
 http://pastebin.com/raw.php?i=cYf8g5tg
 

So this is the issue:

2014-10-20 23:24:27,323 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Seq 1-1609755391808241885:  { Ans: ,
MgmtId: 270567360573239, via: 1, Ver: v1, Flags: 10,
[{com.cloud.agent.api.Answer:{result:false,details:No enum
constant
com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none,wait:0}},{com.cloud.agent.api.Answer:{result:false,details:Stopped
by previous failure,wait:0}}] }

That should be fixed in 4.4.1

By this commit:
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=eeac81838fcc3c53d2115074c8141c9411d3ef4c

Just upgrading the agent to 4.4.1 is enough, the mgmt server can still
run 4.4.0

Wido

 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro
 
 - Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:29:04
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to get answer that is of class
 com.cloud.agent.api.StartAnswer
 
 On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,

 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it 
 has
 worked, except the last step which failed to restart my system VMs. Now I'm 
 on
 4.4.0 without any system VMs; they are stuck in a creation loop which never
 succeeds. Relevant management log:

 Job failed due to exception Resource [Host:1] is unreachable: Host 1: 
 Unable to
 start instance due to Unable to get answer that is of class
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU

 And on the agent:

 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null)
 Processing command: com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , then
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'

 And this goes on and on.

 Any ideas?


 Is all the logging on DEBUG on the Agent? there has to be more
 information in the logfile telling you what is going on exactly.

 Wido

 Lucian


 --
 Sent from the Delta quadrant using Borg technology!

 Nux!
 www.nux.ro

Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-21 Thread Nux! 
Thanks Wido, I'll attempt another upgrade tonight. Cheers!

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Tuesday, 21 October, 2014 08:12:23
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to get answer that is of class
 com.cloud.agent.api.StartAnswer

 On 10/21/2014 12:36 AM, Nux! wrote:
 Wido,
 
 Alas DEBUG is on without telling me anything I can understand. Here's some 
 more:
 
 http://pastebin.com/raw.php?i=cYf8g5tg
 
 
 So this is the issue:
 
 2014-10-20 23:24:27,323 DEBUG [cloud.agent.Agent]
 (agentRequest-Handler-1:null) Seq 1-1609755391808241885:  { Ans: ,
 MgmtId: 270567360573239, via: 1, Ver: v1, Flags: 10,
 [{com.cloud.agent.api.Answer:{result:false,details:No enum
 constant
 com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.diskCacheMode.none,wait:0}},{com.cloud.agent.api.Answer:{result:false,details:Stopped
 by previous failure,wait:0}}] }
 
 That should be fixed in 4.4.1
 
 By this commit:
 https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commitdiff;h=eeac81838fcc3c53d2115074c8141c9411d3ef4c
 
 Just upgrading the agent to 4.4.1 is enough, the mgmt server can still
 run 4.4.0
 
 Wido
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro
 
 - Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:29:04
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to
 get answer that is of class
 com.cloud.agent.api.StartAnswer
 
 On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,

 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it 
 has
 worked, except the last step which failed to restart my system VMs. Now 
 I'm on
 4.4.0 without any system VMs; they are stuck in a creation loop which never
 succeeds. Relevant management log:

 Job failed due to exception Resource [Host:1] is unreachable: Host 1: 
 Unable to
 start instance due to Unable to get answer that is of class
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU

 And on the agent:

 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null)
 Processing command: com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , 
 then
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'

 And this goes on and on.

 Any ideas?


 Is all the logging on DEBUG on the Agent? there has to be more
 information in the logfile telling you what is going on exactly.

 Wido

 Lucian


 --
 Sent from the Delta quadrant using Borg technology!

 Nux!
  www.nux.ro

Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-20 Thread Wido den Hollander 


On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,
 
 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it has 
 worked, except the last step which failed to restart my system VMs. Now I'm 
 on 4.4.0 without any system VMs; they are stuck in a creation loop which 
 never succeeds. Relevant management log:
 
 Job failed due to exception Resource [Host:1] is unreachable: Host 1: Unable 
 to start instance due to Unable to get answer that is of class 
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU
 
 And on the agent:
 
 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null) Processing command: 
 com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection] 
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM, 
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection] 
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM, 
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection] 
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , then 
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource] 
 (agentRequest-Handler-5:null) Failed to get dom xml: 
 org.libvirt.LibvirtException: Domain not found: no domain with matching name 
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource] 
 (agentRequest-Handler-5:null) Failed to get dom xml: 
 org.libvirt.LibvirtException: Domain not found: no domain with matching name 
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource] 
 (agentRequest-Handler-5:null) Failed to get dom xml: 
 org.libvirt.LibvirtException: Domain not found: no domain with matching name 
 'v-439-VM'
 
 And this goes on and on.
 
 Any ideas?
 

Is all the logging on DEBUG on the Agent? there has to be more
information in the logfile telling you what is going on exactly.

Wido

 Lucian
 
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro

Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-20 Thread Nux! 
Wido,

Alas DEBUG is on without telling me anything I can understand. Here's some more:

http://pastebin.com/raw.php?i=cYf8g5tg

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:29:04
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to get answer that is of class
 com.cloud.agent.api.StartAnswer

 On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,
 
 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it has
 worked, except the last step which failed to restart my system VMs. Now I'm 
 on
 4.4.0 without any system VMs; they are stuck in a creation loop which never
 succeeds. Relevant management log:
 
 Job failed due to exception Resource [Host:1] is unreachable: Host 1: Unable 
 to
 start instance due to Unable to get answer that is of class
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU
 
 And on the agent:
 
 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null)
 Processing command: com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , then
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 
 And this goes on and on.
 
 Any ideas?
 
 
 Is all the logging on DEBUG on the Agent? there has to be more
 information in the logfile telling you what is going on exactly.
 
 Wido
 
 Lucian
 
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro

Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-20 Thread Nux! 
I think the problem may be with:

2014-10-20 23:47:57,319 DEBUG [kvm.resource.LibvirtConnection] 
(agentRequest-Handler-3:null) can't find connection: KVM, for vm: v-439-VM, 
continue

What would this mean in English?

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
 From: Nux! n...@li.nux.ro
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:36:49
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to get answer that is of class
 com.cloud.agent.api.StartAnswer

 Wido,
 
 Alas DEBUG is on without telling me anything I can understand. Here's some 
 more:
 
 http://pastebin.com/raw.php?i=cYf8g5tg
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro
 
 - Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:29:04
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to
 get answer that is of class
 com.cloud.agent.api.StartAnswer
 
 On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,
 
 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it 
 has
 worked, except the last step which failed to restart my system VMs. Now I'm 
 on
 4.4.0 without any system VMs; they are stuck in a creation loop which never
 succeeds. Relevant management log:
 
 Job failed due to exception Resource [Host:1] is unreachable: Host 1: 
 Unable to
 start instance due to Unable to get answer that is of class
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU
 
 And on the agent:
 
 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null)
 Processing command: com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , then
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching name
 'v-439-VM'
 
 And this goes on and on.
 
 Any ideas?
 
 
 Is all the logging on DEBUG on the Agent? there has to be more
 information in the logfile telling you what is going on exactly.
 
 Wido
 
 Lucian
 
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
  www.nux.ro

Re: failed 430-440 upgrade: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

2014-10-20 Thread Nux! 
Well, reverted my upgrade and went for 4.3.1 which looks good, will try again 
with 4.4.1 tomorrow night.

Cheers for the help

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
 From: Nux! n...@li.nux.ro
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:57:32
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to get answer that is of class
 com.cloud.agent.api.StartAnswer

 I think the problem may be with:
 
 2014-10-20 23:47:57,319 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-3:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 
 What would this mean in English?
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro
 
 - Original Message -
 From: Nux! n...@li.nux.ro
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:36:49
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to
 get answer that is of class
 com.cloud.agent.api.StartAnswer
 
 Wido,
 
 Alas DEBUG is on without telling me anything I can understand. Here's some 
 more:
 
 http://pastebin.com/raw.php?i=cYf8g5tg
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
 www.nux.ro
 
 - Original Message -
 From: Wido den Hollander w...@widodh.nl
 To: dev@cloudstack.apache.org
 Sent: Monday, 20 October, 2014 23:29:04
 Subject: Re: failed 430-440 upgrade: Unable to start instance due to Unable 
 to
 get answer that is of class
 com.cloud.agent.api.StartAnswer
 
 On 10/21/2014 12:25 AM, Nux! wrote:
 Hi guys,
 
 I've followed the 4.3.x to 4.4.0 upgrade instruction to the letter and it 
 has
 worked, except the last step which failed to restart my system VMs. Now 
 I'm on
 4.4.0 without any system VMs; they are stuck in a creation loop which never
 succeeds. Relevant management log:
 
 Job failed due to exception Resource [Host:1] is unreachable: Host 1: 
 Unable to
 start instance due to Unable to get answer that is of class
 com.cloud.agent.api.StartAnswer
 http://pastebin.com/raw.php?i=k6WksLcU
 
 And on the agent:
 
 2014-10-20 23:10:57,369 DEBUG [cloud.agent.Agent] 
 (agentRequest-Handler-5:null)
 Processing command: com.cloud.agent.api.StopCommand
 2014-10-20 23:10:57,371 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: KVM, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find connection: LXC, for vm: v-439-VM,
 continue
 2014-10-20 23:10:57,372 DEBUG [kvm.resource.LibvirtConnection]
 (agentRequest-Handler-5:null) can't find which hypervisor the vm used , 
 then
 use the default hypervisor
 2014-10-20 23:10:57,374 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'
 2014-10-20 23:10:57,375 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'
 2014-10-20 23:10:57,377 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-5:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching 
 name
 'v-439-VM'
 
 And this goes on and on.
 
 Any ideas?
 
 
 Is all the logging on DEBUG on the Agent? there has to be more
 information in the logfile telling you what is going on exactly.
 
 Wido
 
 Lucian
 
 
 --
 Sent from the Delta quadrant using Borg technology!
 
 Nux!
   www.nux.ro