onitake opened a new issue #3518: Unhandled NPE in VMSnapShotManagerImpl.orchestrateCreateVMSnapshot produces snapshots that cannot be deleted URL: https://github.com/apache/cloudstack/issues/3518 <!-- Verify first that your issue/request is not already reported on GitHub. Also test if the latest release and master branch are affected too. Always add information AFTER of these HTML comments, but no need to delete the comments. --> ##### ISSUE TYPE <!-- Pick one below and delete the rest --> * Bug Report ##### COMPONENT NAME <!-- Categorize the issue, e.g. API, VR, VPN, UI, etc. --> ~~~ Server ~~~ ##### CLOUDSTACK VERSION <!-- New line separated list of affected versions, commit ID for issues on master branch. --> ~~~ at least since 4.11, still present in master ~~~ ##### CONFIGURATION <!-- Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise --> N/A ##### OS / ENVIRONMENT <!-- Information about the environment if relevant, N/A otherwise --> VMware ESX 6.5 CentOS 7 ##### SUMMARY <!-- Explain the problem/feature briefly --> Under certain conditions (races or similar), CloudStack may bails out in the middle of a snapshot creation process. Due to incorrect handling of this condition, a stray snapshot entry may be left in the database, even though no snapshot exists. When trying to delete this bogus entry, another error is thrown (because the snapshot doesn't actually exist) and the entry stays in state "Expunged" indefinitely. I have traced the source of the problem to the `return null` in https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java#L516-L519 This causes a NullPointerException in https://github.com/apache/cloudstack/blob/master/server/src/main/java/com/cloud/vm/snapshot/VMSnapshotManagerImpl.java#L1197-L1199 when `getId()` is called. In my opinion, `orchestrateCreateVMSnapshot` should never return null, but rethrow the exception instead. This would cause the problem to be handled properly later, and an error message returned to the client. It's unclear, however, under which circumstances `takeVMSnapshot` would throw an exception exactly, and if cleanup may be needed. In my opinion, such cleanup should happen right in `takeVMSnapshort`, not be forgotten later. ##### STEPS TO REPRODUCE <!-- For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate. For new features, show how the feature would be used. --> N/A, because we don't know the exact reason why the snapshot creation process may fail. The error printed in the log is as follows: ~~~ 2019-07-24 09:58:04,419 ERROR [cloud.api.ApiAsyncJobDispatcher] (API-Job-Executor-64:ctx-8c081a18 job-2196835) Unexpected exception while executing org.apache.cloudstack.api.command.user.vmsnapshot.CreateVMSnapshotCmd java.lang.RuntimeException: Unexpected exception at com.cloud.vm.snapshot.VMSnapshotManagerImpl.createVMSnapshot(VMSnapshotManagerImpl.java:482) at sun.reflect.GeneratedMethodAccessor817.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy198.createVMSnapshot(Unknown Source) at org.apache.cloudstack.api.command.user.vmsnapshot.CreateVMSnapshotCmd.execute(CreateVMSnapshotCmd.java:113) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at com.cloud.vm.snapshot.VMSnapshotManagerImpl.orchestrateCreateVMSnapshot(VMSnapshotManagerImpl.java:1199) at sun.reflect.GeneratedMethodAccessor818.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ... 26 more 2019-07-24 09:58:04,420 DEBUG [jobs.impl.AsyncJobManagerImpl] (API-Job-Executor-64:ctx-8c081a18 job-2196835) Complete async job-2196835, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unexpected exception"} ~~~ ##### EXPECTED RESULTS <!-- What did you expect to happen when running the steps above? --> If such an error occurs, it should be handled properly and no stray snapshot entry should stay in the database. ##### ACTUAL RESULTS <!-- What actually happened? --> A snapshot entry is created that cannot be deleted.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
