[
https://issues.apache.org/jira/browse/CLOUDSTACK-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13869424#comment-13869424
]
Murali Reddy commented on CLOUDSTACK-5849:
------------------------------------------
Here is my analysis of the bug after looking at the logs.
Network failed to implement in 3.0.6 deployment, because of the error:
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: There are no NetScaler load balancer devices with the free
capacity for implementing this network
at
com.cloud.network.element.NetscalerElement.implement(NetscalerElement.java:176)
So there is NetScaler allocated for the network and subsequently there is no
referance for the network in 'network_external_lb_device_map' as well. Network
went into 'Shutdown' state as expected. There is know issue in 3.0.* releases
where network in 'Shutdown' state can not be deleted which is why there is
error:
2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-70:job-75) Network is not not in the correct state to be
destroyed: Shutdown. Opened 'CLOUDSTACK-5861' to enable force deleting a
network.
Other problem reported in the bug is on upgrade network state changed from
'Shutdown' to 'Implemented', this is a upgrade which needs to be fixed.
> Failed to shutdown the network when corresponding External LB provider gets
> Disabled while still in use by the network
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5849
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5849
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Network Controller, Network Devices
> Affects Versions: 4.3.0
> Environment: upgraded from 3.0.6 patch E to 4.3
> Reporter: manasaveloori
> Assignee: Murali Reddy
> Fix For: 4.3.0
>
> Attachments: management-server306.log.rar, management-server4.3.rar,
> mysqldump306PatchE.dmp, mysqldump4.3.dmp
>
>
> Steps:
> 1. Deployed CS 3.0.6 Patch E with Xen 6.0.2 HV.
> 2. Created the service offering for netscaler device.
> 3. Added incompatible version of Netscaler. added NS10.1: Build 120.1316
> to CS 3.0.6 build.
> it was success.
> 4. Now created the network using the service offering created in step 2.
> 5. Observed that the network went into shut down state as the netscaler
> failed to implement the network.
> 6. Removed the netscaler device from CS and disabled the service offering.
> 7. Now there were no VMs associated to that network.Tried to delete the
> network.But it failed as the network was not in correct state.
> Note: issue existed in 3.0.6 that network is not deleted if it is in shutdown
> state.
> 2014-01-07 21:32:09,720 DEBUG [agent.manager.AgentManagerImpl]
> (RouterMonitor-1:null) Details from executing class
> com.cloud.agent.api.NetworkUsageCommand:
> 2014-01-07 21:32:09,720 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null)
> Recieved and Sent bytes are both 0. Not updating user_statistics
> 2014-01-07 21:32:11,498 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-5:null) Ping from 8
> 2014-01-07 21:32:11,917 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-6:null) Ping from 5
> 2014-01-07 21:32:12,164 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-19:null) submit async job-75, job: AsyncJobVO {id:75, userId:
> 2, accountId: 2, sessionKey: null, instanceType: null, instanceId: null, cmd:
> com.cloud.api.commands.DeleteNetworkCmd, cmdOriginator: null, cmdVersion: 0,
> callbackType: 0, callbackAddress: null, status: 0, processStatus: 0,
> resultCode: 0, result: null, initMsid: 6642334695485, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2014-01-07 21:32:12,169 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd
> for job-75
> 2014-01-07 21:32:12,179 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Sync job-75 execution on object network.216
> 2014-01-07 21:32:12,188 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) job com.cloud.api.commands.DeleteNetworkCmd for
> job-75 was queued, processing the queue.
> 2014-01-07 21:32:12,197 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Executing sync queue item: SyncQueueItemVO {id:17,
> queueId: 17, contentType: AsyncJob, contentId: 75, lastProcessMsid:
> 6642334695485, lastprocessNumber: 1, lastProcessTime: Tue Jan 07 21:32:12 IST
> 2014, created: Tue Jan 07 21:32:12 IST 2014}
> 2014-01-07 21:32:12,199 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-69:job-75) Schedule queued job-75
> 2014-01-07 21:32:12,207 DEBUG [cloud.async.SyncQueueManagerImpl]
> (Job-Executor-69:job-75) There is a pending process in sync queue(id: 17)
> 2014-01-07 21:32:12,210 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-70:job-75) Executing com.cloud.api.commands.DeleteNetworkCmd
> for job-75
> 2014-01-07 21:32:12,233 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-70:job-75) Network is not implemented: Ntwk[216|Guest|17]
> 2014-01-07 21:32:12,233 DEBUG [db.Transaction.Transaction]
> (Job-Executor-70:job-75) Rolling back the transaction: Time = 1 Name =
> -AsyncJobManagerImpl$1.run:396-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679;
> called by
> -Transaction.rollback:854-Transaction.removeUpTo:797-Transaction.close:621-DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-NetworkManagerImpl.destroyNetwork:3799-DatabaseCallback.intercept:30-NetworkManagerImpl.deleteNetwork:3656-ActionEventCallback.intercept:32-DeleteNetworkCmd.execute:65-ApiDispatcher.dispatch:263-AsyncJobManagerImpl$1.run:430
> 2014-01-07 21:32:12,235 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-70:job-75) Network is not not in the correct state to be
> destroyed: Shutdown
> 2014-01-07 21:32:12,242 WARN [cloud.api.ApiDispatcher]
> (Job-Executor-70:job-75) class com.cloud.api.ServerApiException : Failed to
> delete network
> 2014-01-07 21:32:12,243 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-70:job-75) Complete async job-75, jobStatus: 2, resultCode:
> 530, result: Error Code: 530 Error text: Failed to delete network
> 8. Now upgraded the CS to 4.3.
> 9. The network state changed to implemented.
> 10.Tried to delete the network now which failed.Following Error is observed:
> 2014-01-09 22:51:18,420 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-25:ctx-20a7144b) ===START=== 10.252.192.34 -- GET
> command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) submit async job-217, details:
> AsyncJobVO {id:217, userId: 2, accountId: 2, instanceType: None, instanceId:
> null, cmd: org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> cmdInfo:
> {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-01-09 22:51:18,461 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-25:ctx-20a7144b ctx-bbdb1ebe) ===END=== 10.252.192.34 -- GET
> command=deleteNetwork&id=037f906f-3a80-4624-a036-9f68f27514fa&response=json&sessionkey=2jZO1DYoLAMU6TkyWkYScSpPZyw%3D&_=1389268728955
> 2014-01-09 22:51:18,463 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-39:ctx-a33dc0eb) Add job-217 into job monitoring
> 2014-01-09 22:51:18,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Executing AsyncJobVO {id:217, userId: 2,
> accountId: 2, instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> {"response":"json","id":"037f906f-3a80-4624-a036-9f68f27514fa","sessionkey":"2jZO1DYoLAMU6TkyWkYScSpPZyw\u003d","cmdEventType":"NETWORK.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1389268728955","ctxAccountId":"2","ctxStartEventId":"786"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6642334695485, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-01-09 22:51:18,504 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is acquired for network
> Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,516 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 port forwarding rules
> for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,516 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,518 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 static nat rules for
> network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,519 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Public load balancing
> rules for network id=217
> 2014-01-09 22:51:18,521 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
> to forward to the network elements
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Revoking 0 Internal load
> balancing rules for network id=217
> 2014-01-09 22:51:18,523 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no Load Balancing Rules
> to forward to the network elements
> 2014-01-09 22:51:18,525 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall ingress
> rules for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,525 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,527 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Releasing 0 firewall egress rules
> for network id=217 as a part of shutdownNetworkRules
> 2014-01-09 22:51:18,533 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) applying default firewall egress
> rules
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.e.VirtualRouterElement]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't
> need to apply firewall rules on the backend; virtual router doesn't exist in
> the network 217
> 2014-01-09 22:51:18,553 DEBUG [c.c.n.f.FirewallManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) There are no rules to forward to
> the network elements
> 2014-01-09 22:51:18,555 DEBUG [c.c.n.r.RulesManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Found 0 static nat rules to apply
> for network id 217
> 2014-01-09 22:51:18,575 DEBUG [c.c.n.e.VirtualRouterElement]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Virtual router elemnt doesn't
> need to associate ip addresses on the backend; virtual router doesn't exist
> in the network 217
> 2014-01-09 22:51:18,583 WARN [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Unable to complete shutdown of
> the network elements due to element: Netscaler either doesn't exist or not
> enabled in the physical network 200
> 2014-01-09 22:51:18,583 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
> Netscaler
> 2014-01-09 22:51:18,588 WARN [c.c.n.ExternalLoadBalancerDeviceManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network shutdwon requested on
> external load balancer element, which did not implement the network. Either
> network implement failed half way through or already network shutdown is
> completed. So just returning.
> 2014-01-09 22:51:18,593 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Sending network shutdown to
> VirtualRouter
> 2014-01-09 22:51:18,604 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Lock is released for network
> Ntwk[217|Guest|17] as a part of network shutdown
> 2014-01-09 22:51:18,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (Job-Executor-39:ctx-a33dc0eb ctx-bbdb1ebe) Network is not not in the correct
> state to be destroyed: Implemented
> 2014-01-09 22:51:18,619 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Complete async job-217, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to delete network"}
> 2014-01-09 22:51:18,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-39:ctx-a33dc0eb) Done executing
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-217
> Attaching the MS ,DB dumps.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
