andrijapanicsb commented on issue #3264: Volume snapshot listing failing when originating primary pool removed URL: https://github.com/apache/cloudstack/issues/3264#issuecomment-583518558 @shwstppr @rhtyd The issue is still present and reproducible on both 4.11.2 and master - just tested. 1. create snap of a volume 2. notice on WHICH primary storage the volume (not the snap) is located (i.e. PRI2) 3. go to DB and set the removed date on "removed" column/field in storage_pool table 4. list snapshots via CMK and observer the issue: **Before "deleting" storage:** > (localcloud) SBCM5> > list snapshots id=1a63b5d5-25ad-41c4-80c7-388050cbaf47 { "count": 1, "snapshot": [ { "account": "admin", "created": "2020-02-07T17:26:56+0000", "domain": "ROOT", "domainid": "37bbcc53-4980-11ea-952a-1e00b30107e1", "id": "1a63b5d5-25ad-41c4-80c7-388050cbaf47", "intervaltype": "MANUAL", "jobid": "0c23606f-7dca-463b-b0dd-39bb59ed541e", "jobstatus": 0, "name": "snap-kvm-andrija1", "osdisplayname": "CentOS 5.5 (64-bit)", "ostypeid": "37f05328-4980-11ea-952a-1e00b30107e1", "physicalsize": 0, "revertable": true, "snapshottype": "MANUAL", "state": "BackingUp", "tags": [], "virtualsize": 8589934592, "volumeid": "c1f9abbd-61c6-41c8-a8b2-cc04767fd557", "volumename": "ROOT-8", "volumetype": "ROOT", "zoneid": "6bd91d12-cad9-477b-9300-27678ce9e80f" } ] } After "deleting" the storage: > (localcloud) SBCM5> > list snapshots id=1a63b5d5-25ad-41c4-80c7-388050cbaf47 Error: (HTTP 530, error code 9999) <nil> And the same exception in logs: > 2020-02-07 17:29:46,777 DEBUG [c.c.a.ApiServlet] (qtp504527234-13:ctx-f0fb97c1) (logid:507c01c5) ===START=== 10.2.2.233 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listSnapshots&id=1a63b5d5-25ad-41c4-80c7-388050cbaf47&response=json&signature=7d%2FoaRsxGhOx1vLpqKZULO9aFTc%3D 2020-02-07 17:29:46,785 DEBUG [c.c.a.ApiServer] (qtp504527234-13:ctx-f0fb97c1 ctx-84cdffe8) (logid:507c01c5) CIDRs from which account 'Acct[70becd07-4980-11ea-952a-1e00b30107e1-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2020-02-07 17:29:46,803 ERROR [c.c.a.ApiServer] (qtp504527234-13:ctx-f0fb97c1 ctx-84cdffe8 ctx-337c9a51) (logid:507c01c5) unhandled exception executing api command: [Ljava.lang.String;@772bb72a java.lang.NullPointerException at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:982) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69) at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:134) at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:552) at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:759) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:583) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89) at javax.servlet.http.HttpServlet.service(HttpServlet.java:686) at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:530) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 2020-02-07 17:29:46,806 DEBUG [c.c.a.ApiServlet] (qtp504527234-13:ctx-f0fb97c1 ctx-84cdffe8 ctx-337c9a51) (logid:507c01c5) ===END=== 10.2.2.233 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listSnapshots&id=1a63b5d5-25ad-41c4-80c7-388050cbaf47&response=json&signature=7d%2FoaRsxGhOx1vLpqKZULO9aFTc%3D 2020-02-07 17:29:48,072 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) SeqA 7-5096: Processing Seq 7-5096: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2020-02-07 17:29:48,076 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) SeqA 7-5096: Sending Seq 7-5096: { Ans: , MgmtId: 32988352022497, via: 7, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2020-02-07 17:29:48,400 DEBUG [c.c.a.ApiServlet] (qtp504527234-20:ctx-2b2cb631) (logid:56a35fa0) ===START=== 10.2.2.233 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listSnapshots&response=json&signature=Ko6ONMB%2FT3U3UQdfiytAXUOGcjw%3D 2020-02-07 17:29:48,405 DEBUG [c.c.a.ApiServer] (qtp504527234-20:ctx-2b2cb631 ctx-8e2267f0) (logid:56a35fa0) CIDRs from which account 'Acct[70becd07-4980-11ea-952a-1e00b30107e1-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2020-02-07 17:29:48,421 ERROR [c.c.a.ApiServer] (qtp504527234-20:ctx-2b2cb631 ctx-8e2267f0 ctx-4b6cab92) (logid:56a35fa0) unhandled exception executing api command: [Ljava.lang.String;@5fc1e40f java.lang.NullPointerException at org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:982) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95) at org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69) at org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:134) at com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:552) at org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:759) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:583) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89) at javax.servlet.http.HttpServlet.service(HttpServlet.java:686) at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:530) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 2020-02-07 17:29:48,433 DEBUG [c.c.a.ApiServlet] (qtp504527234-20:ctx-2b2cb631 ctx-8e2267f0 ctx-4b6cab92) (logid:56a35fa0) ===END=== 10.2.2.233 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listSnapshots&response=json&signature=Ko6ONMB%2FT3U3UQdfiytAXUOGcjw%3D
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
