[
https://issues.apache.org/jira/browse/CLOUDSTACK-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sheng Yang resolved CLOUDSTACK-689.
-----------------------------------
Resolution: Fixed
Fixed at:
commit 53e6825b744252daeab7c4e694adae69c65cf0d3
Author: Sheng Yang <[email protected]>
Date: Thu Dec 20 17:01:12 2012 -0800
CLOUDSTACK-689: Clean stop pending flags when VR start/stop
> RVR: Stop pending flag is not cleared when user start the disconnected router
> from another host
> -----------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-689
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-689
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Network Controller
> Affects Versions: 4.0.0
> Reporter: Sheng Yang
> Assignee: Sheng Yang
> Fix For: 4.1.0
>
>
> Here is what's happened:
> 2012-09-16 21:27:24,285 WARN
> [network.router.VirtualNetworkApplianceManagerImpl] (HA-Worker-0:work-37779)
> Unable to apply dhcp entry on disconnected router r-38445-VM
> com.cloud.exception.AgentUnavailableException: Resource [Host:610] is
> unreachable: Host 610: Host is not in the right state: Down
> [...]
> 2012-09-16 21:27:24,286 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl] (HA-Worker-0:work-37779)
> About to stop the router r-38445-VM due to: Unable to apply dhcp entry on
> disconnected router
> At the above time, we set stop pending flag, due to host 610 is down, and we
> failed to reach the router r-38445-VM.
> Then:
> 2012-09-16 21:51:54,067 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-28:null) submit async job-296045, details: AsyncJobVO
> {id:296045, userId: 6217, accountId: 6178, sessionKey: null, instanceType:
> DomainRouter, instanceId: 38445, cmd: com.cloud.api.commands.StartRouterCmd,
> cmdOriginator: null, cmdInfo:
> {"response":"json","id":"f14313a7-5490-408f-a9d7-292303784f8f","sessionkey":"lByhfEpE7fygtOASvMH3i8xgEok\u003d","ctxUserId":"6217","_":"1347799961307","ctxAccountId":"6178","ctxStartEventId":"1533940"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 152687106451262,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2012-09-16 21:51:54,070 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-126:job-296045) Executing com.cloud.api.commands.StartRouterCmd
> for job-296045
> User start the router manually, and CS deploy it on host 606:
> 2012-09-16 21:51:56,069 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-126:job-296045) Deployment found -
> P0=VM[DomainRouter|r-38445-VM],
> P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(
> 2)-Pod(29)-Cluster(52)-Host(606)-Storage()]
> 2012-09-16 21:51:56,090 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-126:job-296045) VM state transitted from :Starting to Starting
> with event: OperationRetryvm's original host id: 610 new host id: 606 host id
> before state transiti
> on: null
> We didn't cover this situation in our code. And this won't clear stop pending
> flag of course.
> Also the stop pending flag can be set if the command sent to router is time
> out, but host is not detected as Disconnected by CS.
> See the mgmt log attached for r-5-VM:
> 2012-09-29 22:20:55,413 DEBUG [agent.manager.AgentAttache]
> (Job-Executor-130:job-450) Seq 1-1662189781: Cancelling.
> 2012-09-29 22:20:55,413 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-130:job-450) Timed Out
> com.cloud.exception.OperationTimedoutException: Commands 1662189781 to Host 1
> timed out after 3600
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:425)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:501)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1022)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2590)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendPortForwardingRules(VirtualNetworkApplianceManagerImpl.java:2725)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$5.execute(VirtualNetworkApplianceManagerImpl.java:2703)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:2809)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:2688)
>
> at
> com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:636)
>
> at
> com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3245)
> at
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:374)
>
> at
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRulesForNetwork(RulesManagerImpl.java:714)
>
> at
> com.cloud.network.NetworkManagerImpl.reprogramNetworkRules(NetworkManagerImpl.java:3516)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1812)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1754)
>
> at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
> at
> com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:3436)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2285)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2255)
>
> at
> com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:32)
> at
> com.cloud.api.commands.StartRouterCmd.execute(StartRouterCmd.java:101)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:427)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:636)
> 2012-09-29 22:20:55,414 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-130:job-450) Unable to apply firewall rules on disconnected
> router r-5-VM
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to send commands to virtual router
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2593)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendPortForwardingRules(VirtualNetworkApplianceManagerImpl.java:2725)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$5.execute(VirtualNetworkApplianceManagerImpl.java:2703)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:2809)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:2688)
>
> at
> com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:636)
>
> at
> com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:3245)
> at
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:374)
>
> at
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRulesForNetwork(RulesManagerImpl.java:714)
>
> at
> com.cloud.network.NetworkManagerImpl.reprogramNetworkRules(NetworkManagerImpl.java:3516)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1812)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1754)
>
> at
> com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
> at
> com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:3436)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2285)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2255)
>
> at
> com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:32)
> at
> com.cloud.api.commands.StartRouterCmd.execute(StartRouterCmd.java:101)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:427)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:636)
> Caused by: com.cloud.exception.OperationTimedoutException: Commands
> 1662189781 to Host 1 timed out after 3600
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:425)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:501)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:1022)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:2590)
>
> ... 33 more
> 2012-09-29 22:20:55,415 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-130:job-450) About to stop the router r-5-VM due to: Unable to
> apply firewall rules on disconnected router
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
