[
https://issues.apache.org/jira/browse/CLOUDSTACK-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roeland Kuipers updated CLOUDSTACK-4606:
----------------------------------------
Description:
If a routing VM hangs on kernel panic. It will never be recovered by the HA
process of Cloudstack.
In this case the Virtual Router cannot be accessed anymore over link-local
and/or management address and cannot be managed anymore.
We think HA should reboot a router when this occurs.
See also CLOUDSTACK-4607 and CLOUDSTACK-4606
Errors observed while hanging:
013-09-04 18:57:42,108 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-446:null) callHostPlugin failed for cmd: routerProxy with args
args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
create input stream: Read timed out
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g,
due to Failed to create input stream: Read timed out
2013-09-04 18:57:42,109 DEBUG [agent.transport.Request] (DirectAgent-446:null)
Seq 105-1580215291: Processing: { Ans: , MgmtId: 345052370017, via: 105, Ver:
v1, Flags: 10, [{"NetworkUsageAnswer":{"result":false,"details":"Exception:
com.cloud.utils.exception.CloudRuntimeException\nMessage: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed
out\nStack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin
failed for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out\n\tat
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3745)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.VPCNetworkUsage(XenServer56Resource.java:186)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.execute(XenServer56Resource.java:211)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:71)\n\tat
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)\n\tat
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:138)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\n\tat
java.lang.Thread.run(Thread.java:662)\n","wait":0}}] }
Message: callHostPlugin failed for cmd: routerProxy with args args:
vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
input stream: Read timed out
Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out
Message: callHostPlugin failed for cmd: routerProxy with args args:
vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
input stream: Read timed out
Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out
2013-09-04 19:08:20,909 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-367:null) callHostPlugin failed for cmd: routerProxy with args
args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
create input stream: Read timed out
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g,
due to Failed to create input stream: Read timed out
was:
If a routing VM hangs on kernel panic. It will never be recovered by the HA
process of Cloudstack.
In this case the Virtual Router cannot be accessed anymore over link-local
and/or management address and cannot be managed anymore.
We think HA should reboot a router when this occurs.
Errors observed while hanging:
013-09-04 18:57:42,108 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-446:null) callHostPlugin failed for cmd: routerProxy with args
args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
create input stream: Read timed out
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g,
due to Failed to create input stream: Read timed out
2013-09-04 18:57:42,109 DEBUG [agent.transport.Request] (DirectAgent-446:null)
Seq 105-1580215291: Processing: { Ans: , MgmtId: 345052370017, via: 105, Ver:
v1, Flags: 10, [{"NetworkUsageAnswer":{"result":false,"details":"Exception:
com.cloud.utils.exception.CloudRuntimeException\nMessage: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed
out\nStack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin
failed for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out\n\tat
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3745)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.VPCNetworkUsage(XenServer56Resource.java:186)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.execute(XenServer56Resource.java:211)\n\tat
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:71)\n\tat
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)\n\tat
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:138)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\n\tat
java.lang.Thread.run(Thread.java:662)\n","wait":0}}] }
Message: callHostPlugin failed for cmd: routerProxy with args args:
vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
input stream: Read timed out
Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out
Message: callHostPlugin failed for cmd: routerProxy with args args:
vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
input stream: Read timed out
Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
95.142.107.111 -g, due to Failed to create input stream: Read timed out
2013-09-04 19:08:20,909 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-367:null) callHostPlugin failed for cmd: routerProxy with args
args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
create input stream: Read timed out
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd:
routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g,
due to Failed to create input stream: Read timed out
> HA does not kick on Routing VM hanging on kernel panic.
> -------------------------------------------------------
>
> Key: CLOUDSTACK-4606
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4606
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Virtual Router
> Affects Versions: 4.1.1
> Reporter: Roeland Kuipers
>
> If a routing VM hangs on kernel panic. It will never be recovered by the HA
> process of Cloudstack.
> In this case the Virtual Router cannot be accessed anymore over link-local
> and/or management address and cannot be managed anymore.
> We think HA should reboot a router when this occurs.
> See also CLOUDSTACK-4607 and CLOUDSTACK-4606
> Errors observed while hanging:
> 013-09-04 18:57:42,108 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-446:null) callHostPlugin failed for cmd: routerProxy with args
> args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
> create input stream: Read timed out
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
> cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed out
> 2013-09-04 18:57:42,109 DEBUG [agent.transport.Request]
> (DirectAgent-446:null) Seq 105-1580215291: Processing: { Ans: , MgmtId:
> 345052370017, via: 105, Ver: v1, Flags: 10,
> [{"NetworkUsageAnswer":{"result":false,"details":"Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: callHostPlugin
> failed for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed
> out\nStack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin
> failed for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed
> out\n\tat
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3745)\n\tat
>
> com.cloud.hypervisor.xen.resource.XenServer56Resource.VPCNetworkUsage(XenServer56Resource.java:186)\n\tat
>
> com.cloud.hypervisor.xen.resource.XenServer56Resource.execute(XenServer56Resource.java:211)\n\tat
>
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:71)\n\tat
>
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)\n\tat
>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)\n\tat
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:138)\n\tat
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)\n\tat
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\n\tat
> java.lang.Thread.run(Thread.java:662)\n","wait":0}}] }
> Message: callHostPlugin failed for cmd: routerProxy with args args:
> vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
> input stream: Read timed out
> Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
> for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed out
> Message: callHostPlugin failed for cmd: routerProxy with args args:
> vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to create
> input stream: Read timed out
> Stack: com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed
> for cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed out
> 2013-09-04 19:08:20,909 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-367:null) callHostPlugin failed for cmd: routerProxy with args
> args: vpc_netusage.sh 169.254.3.166 -l 95.142.107.111 -g, due to Failed to
> create input stream: Read timed out
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
> cmd: routerProxy with args args: vpc_netusage.sh 169.254.3.166 -l
> 95.142.107.111 -g, due to Failed to create input stream: Read timed out
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
