[
https://issues.apache.org/jira/browse/CLOUDSTACK-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594142#comment-13594142
]
Alena Prokharchyk commented on CLOUDSTACK-1368:
-----------------------------------------------
It is a big bug in Network State transitioning introduced with the commit
e7a554fc6a23a49949c2d88d6ef680682c6f6bc4 (Events framework merge). Murali,
reassigning it to you. See the history of the problem and the steps to fix the
below. It should be fixed in both 4.1 and master branches.
History:
* Before 4.1 although the Network state machine was in place (Network.java), it
was never referred in the code, All the state changes were hardcoded in the
NetworkManagerImpl - wrong way to do.
* After bringing up the Event manager code, all state changes for the network
started going through the state machine, which is totally correct and that's
how it should have been implemented from the very beginning. The problem is,
the some state transitions are incorrect, and some are missing. Therefore the
incoming bugs.
Things to fix:
1) Add state transitions for the "Setup" state. That's what's causing this
particular bug to happen - all shared networks come in Setup state by default,
and never go to Implemented state.
2) Change event for the line:
s_fsm.addTransition(State.Implemented, Event.DestroyNetwork, State.Shutdown)
The event should be Event.Shutdown instead of Event.DestroyNetwork.
3) Add state transition for Destroy event. The original state should be
"Shutdown", the target state in success case - 'Destroy', in failure case -
'Shutdown'
4) Fix the comment for the following line (Shutdown states for the network
being Shutdown, not Destroy. Shutdown call can be triggered either by network
GC thread, or by Destroy network call):
"Shutdown("Indicates the network configuration is being destroyed")"
5) Go through all the methods propagating the state change events in
NetworkManagerImpl and see if the proper events are being passed in. "Shutdown"
event should be passed from shutdown call, and "Destroy" event should be passed
only from destroy network.
6) I know that you've added the Shutdown support for the Shared network, so
they can be picked up by the network GC. Make sure that state transitions are
supported for this case (the network comes in Setup state by default)
After putting in changes, QA for both Shared and Isolated network should be
done. Have to test the network lifecycle - create/shutdown (GC)/destroy
networks of both types. Murali, after unittest is passed, please come up with
the list of tests for the QA so all the methods where state machine use was
introduced, are covered. Everything should behave as it used to before the
change.
> Shared network - Not able to delete network because of
> java.lang.NullPointerException.
> ---------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-1368
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1368
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.1.0
> Environment: Latest build from 4.1
> Reporter: Sangeetha Hariharan
> Assignee: Alena Prokharchyk
> Priority: Blocker
> Fix For: 4.1.0
>
>
> Shared network - Not able to delete network because of
> java.lang.NullPointerException.
> Set up:
> Advanced Zone with rhel 6.3 KVM host.
> Create a shared network with dual stack (ipv4 and ipv6).
> Delete this network.
> Delete netwpork fails with "Command failed due to Internal Server Error"
> errotr message:
> 2013-02-21 17:00:19,473 INFO [cloud.api.ApiServer] (catalina-exec-20:null)
> (userId=2 accountId=2 sessionId=8323A1028B9235FCD786CDEAA7535425)
> 10.216.50.215 -- GET command=deleteNetwor
> k&id=b8ebcb5e-f0db-4997-9080-b7156d56bab9&response=json&sessionkey=LUODgzUxTfC7VuYYEVeNQ0oNo2c%3D&_=1361494821669
> 200 { "deletenetworkresponse" : {"jobid":"a2201560-eaf5-4d08-a5a7-afb
> d9e209934"} }
> 2013-02-21 17:00:22,671 INFO [cloud.api.ApiServer] (catalina-exec-17:null)
> (userId=2 accountId=2 sessionId=8323A1028B9235FCD786CDEAA7535425)
> 10.216.50.215 -- GET command=queryAsyncJo
> bResult&jobId=a2201560-eaf5-4d08-a5a7-afbd9e209934&response=json&sessionkey=LUODgzUxTfC7VuYYEVeNQ0oNo2c%3D&_=1361494824961
> 200 { "queryasyncjobresultresponse" : {"accountid":"9bdc3f60
> -7c7d-11e2-89d9-06d4460004b1","userid":"9bdcb7c4-7c7d-11e2-89d9-06d4460004b1","cmd":"org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd","jobstatus":2,"jobprocstatus":0,"
> jobresultcode":530,"jobresulttype":"object","jobresult":{"errorcode":530,"errortext":"Command
> failed due to Internal Server
> Error"},"created":"2013-02-21T17:00:19-0800","jobid":"a2201
> 560-eaf5-4d08-a5a7-afbd9e209934"} }
> Management-server.logs:
> 2013-02-21 17:00:19,472 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-20:null) submit async job-13, details
> : AsyncJobVO {id:13, userId: 2, accountId: 2, sessionKey: null, instanceType:
> null, instanceId: null, cmd: org.apach
> e.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdOriginator: null,
> cmdInfo: {"id":"b8ebcb5e-f0db-4997-9080
> -b7156d56bab9","response":"json","sessionkey":"LUODgzUxTfC7VuYYEVeNQ0oNo2c\u003d","ctxUserId":"2","_":"1361494821669
> ","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0,
> callbackAddress: null, status: 0, proc
> essStatus: 0, resultCode: 0, result: null, initMsid: 7508777239729,
> completeMsid: null, lastUpdated: null, lastPolle
> d: null, created: null}
> 2013-02-21 17:00:19,473 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
> ===END=== 10.216.50.215 -- GET comman
> d=deleteNetwork&id=b8ebcb5e-f0db-4997-9080-b7156d56bab9&response=json&sessionkey=LUODgzUxTfC7VuYYEVeNQ0oNo2c%3D&_=13
> 61494821669
> 2013-02-21 17:00:19,478 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-13:job-13) Executing org.apache.clouds
> tack.api.command.user.network.DeleteNetworkCmd for job-13
> 2013-02-21 17:00:19,514 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-13:job-13) Sync job-13 execution on ob
> ject network.205
> 2013-02-21 17:00:19,547 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-13:job-13) job org.apache.cloudstack.a
> pi.command.user.network.DeleteNetworkCmd for job-13 was queued, processing
> the queue.
> 2013-02-21 17:00:19,555 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-13:job-13) Executing sync queue item:
> SyncQueueItemVO {id:1, queueId: 1, contentType: AsyncJob, contentId: 13,
> lastProcessMsid: 7508777239729, lastprocess
> Number: 1, lastProcessTime: Thu Feb 21 17:00:19 PST 2013, created: Thu Feb 21
> 17:00:19 PST 2013}
> 2013-02-21 17:00:19,556 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-13:job-13) Schedule queued job-13
> 2013-02-21 17:00:19,561 DEBUG [cloud.async.SyncQueueManagerImpl]
> (Job-Executor-13:job-13) There is a pending process
> in sync queue(id: 1)
> 2013-02-21 17:00:19,562 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-14:job-13) Executing org.apache.clouds
> tack.api.command.user.network.DeleteNetworkCmd for job-13
> 2013-02-21 17:00:19,607 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Network is not implemented
> : Ntwk[205|Guest|7]
> 2013-02-21 17:00:19,620 DEBUG [network.rules.RulesManagerImpl]
> (Job-Executor-14:job-13) Releasing 0 port forwarding
> rules for network id=205
> 2013-02-21 17:00:19,621 DEBUG [network.rules.RulesManagerImpl]
> (Job-Executor-14:job-13) Releasing 0 static nat rules
> for network id=205
> 2013-02-21 17:00:19,624 DEBUG [network.rules.RulesManagerImpl]
> (Job-Executor-14:job-13) There are no port forwarding
> rules to apply for network id=205
> 2013-02-21 17:00:19,625 DEBUG [network.rules.RulesManagerImpl]
> (Job-Executor-14:job-13) There are no static nat rule
> s to apply for network id=205
> 2013-02-21 17:00:19,627 DEBUG [network.rules.RulesManagerImpl]
> (Job-Executor-14:job-13) Successfully released rules
> for network id=205 and # of rules now = 0
> 2013-02-21 17:00:19,627 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Successfully cleaned up po
> rtForwarding/staticNat rules for network id=205
> 2013-02-21 17:00:19,636 DEBUG [network.lb.LoadBalancingRulesManagerImpl]
> (Job-Executor-14:job-13) Found 0 lb rules t
> o cleanup
> 2013-02-21 17:00:19,636 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Successfully cleaned up lo
> ad balancing rules for network id=205
> 2013-02-21 17:00:19,637 DEBUG [network.firewall.FirewallManagerImpl]
> (Job-Executor-14:job-13) Releasing 0 firewall r
> ules for network id=205
> 2013-02-21 17:00:19,637 DEBUG [network.firewall.FirewallManagerImpl]
> (Job-Executor-14:job-13) There are no firewall
> rules to apply
> 2013-02-21 17:00:19,638 DEBUG [network.firewall.FirewallManagerImpl]
> (Job-Executor-14:job-13) Successfully released
> firewall rules for network id=205 and # of rules now = 0
> 2013-02-21 17:00:19,638 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Successfully cleaned up fi
> rewallRules rules for network id=205
> 2013-02-21 17:00:19,644 DEBUG [network.vpc.NetworkACLManagerImpl]
> (Job-Executor-14:job-13) Found no network ACLs for
> network id=205
> 2013-02-21 17:00:19,644 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Successfully cleaned up Ne
> tworkACLs for network id=205
> 2013-02-21 17:00:19,652 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Sending destroy to com.clo
> ud.network.element.VirtualRouterElement@71fdbe17
> 2013-02-21 17:00:19,656 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Network id=205 is destroye
> d successfully, cleaning up corresponding resources now.
> 2013-02-21 17:00:19,673 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-14:job-13) Deleted ip range for priva
> te network id=205
> 2013-02-21 17:00:19,673 DEBUG [db.Transaction.Transaction]
> (Job-Executor-14:job-13) Rolling back the transaction: Ti
> me = 17 Name =
> -AsyncJobManagerImpl$1.run:401-Executors$RunnableAdapter.call:471-FutureTask$Sync.innerRun:334-Futur
> eTask.run:166-ThreadPoolExecutor.runWorker:1110-ThreadPoolExecutor$Worker.run:603-Thread.run:679;
> called by -Transac
> tion.rollback:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.AroundAnyMethod:39-Gene
> ratedMethodAccessor35.invoke:-1-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:616-AbstractAspectJAdvice.invok
> eAdviceMethodWithGivenArgs:621-AbstractAspectJAdvice.invokeAdviceMethod:610-AspectJAroundAdvice.invoke:65-Reflective
> MethodInvocation.proceed:172-ExposeInvocationInterceptor.invoke:90
> 2013-02-21 17:00:19,714 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-14:job-13) Unexpected exception while
> executing org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> java.lang.NullPointerException
> at
> com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:81)
> at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:100)
> at
> com.cloud.network.NetworkManagerImpl.stateTransitTo(NetworkManagerImpl.java:3526)
> at
> com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkManagerImpl.java:2246)
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.jav
> a:689)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoi
> nPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
> at
> com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1343)
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.event.ActionEventInterceptor.AroundAnyMethod(ActionEventInterceptor.java:41)
> at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
