Hello Suresh, I deleted the snapshot policies using the UI and recreated them and it seems to fix the issue.
Will the retention limit delete the snapshots from the previous policy? Regards, Antoine > On May 4, 2022, at 3:25 AM, Suresh Anaparti <[email protected]> > wrote: > > Hi Antoine, > > It seems, there are recurring snapshots scheduled on volume id: > 390198b4-45d4-425e-87c3-d0f11ef2ca11. Is that volume still exists and in > Ready state (Is it listed in UI? else, check with listVolumes API). Also, > check the snapshot policies for that volume (from UI if listed, else with API > listSnapshotPolicies using volumeid param) and remove them (from UI, or with > API deleteSnapshotPolicies using id from the listSnapshotPolicies API > response). > > > Regards, > Suresh > > > > > From: Antoine Boucher <[email protected] <mailto:[email protected]>> > Reply to: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Date: Sunday, 1 May 2022 at 5:04 AM > To: users <[email protected]> > Subject: Re: No longer able to see any "Snapshots" > > Hi Suresh, > > Thank you for your reply, Yes this is the local storage of a Hiost that was > removed. It is strange since all VM where migrated before. The host was then > put in maintenance and eventually removed. > > What would be the fix? Removing the snapshot record(s)? > > Is this also related? I’m seeing this events every 5 minutes,.. > > > > ERROR > <https://nimbus.haltondc.com:8443/client/#/event/d8ae2255-0f91-4041-9a19-ecd81bb38244> > > SNAPSHOT.CREATE > Created > Error while creating entity for allocating snapshot > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838> > > system > <https://nimbus.haltondc.com:8443/client/#/account?name=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838&dataView=true> > > ROOT > <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838> > > 30 Apr 2022 19:32:41 > INFO > <https://nimbus.haltondc.com:8443/client/#/event/50ea9c36-745d-4a44-935e-551f6294e439> > > SNAPSHOT.CREATE > Scheduled > creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11 > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > Networth-Abrams > <https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11&dataView=true> > > Networth > <https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > 30 Apr 2022 19:32:41 > ERROR > <https://nimbus.haltondc.com:8443/client/#/event/40bd9bc2-5899-49ff-9833-fb29dea7c0ba> > > SNAPSHOT.CREATE > Created > Error while creating entity for allocating snapshot > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838> > > system > <https://nimbus.haltondc.com:8443/client/#/account?name=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838&dataView=true> > > ROOT > <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838> > > 30 Apr 2022 19:27:41 > INFO > <https://nimbus.haltondc.com:8443/client/#/event/1dce5b57-4dbe-4201-b04a-6dfa63363014> > > SNAPSHOT.CREATE > Scheduled > creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11 > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > Networth-Abrams > <https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11&dataView=true> > > Networth > <https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > 30 Apr 2022 19:27:41 > ERROR > <https://nimbus.haltondc.com:8443/client/#/event/3c443f26-0e93-4f9e-9d69-192794c1efec> > > SNAPSHOT.CREATE > Created > Error while creating entity for allocating snapshot > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838> > > system > <https://nimbus.haltondc.com:8443/client/#/account?name=system&domainid=088d13e7-8bc4-11ec-9c43-001e67fd4838&dataView=true> > > ROOT > <https://nimbus.haltondc.com:8443/client/#/domain/088d13e7-8bc4-11ec-9c43-001e67fd4838> > > 30 Apr 2022 19:22:41 > INFO > <https://nimbus.haltondc.com:8443/client/#/event/ca2e5d1a-9ad1-4f9f-9d5a-45f3e9ec42ef> > > SNAPSHOT.CREATE > Scheduled > creating snapshot for volume Id:390198b4-45d4-425e-87c3-d0f11ef2ca11 > system > <https://nimbus.haltondc.com:8443/client/#/accountuser?username=system&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > Networth-Abrams > <https://nimbus.haltondc.com:8443/client/#/account?name=Networth-Abrams&domainid=80dc2363-7b2b-4614-a02f-b0e5a1782b11&dataView=true> > > Networth > <https://nimbus.haltondc.com:8443/client/#/domain/80dc2363-7b2b-4614-a02f-b0e5a1782b11> > > 30 Apr 2022 19:22:41 > > > > > Regards, > Antoine > > > > Antoine Boucher > [email protected] <mailto:[email protected]> > [o] +1-226-505-9734 > www.haltondc.com <http://www.haltondc.com/> > > “Data security made simple and affordable” > > > > > > > > Confidentiality Warning: This message and any attachments are intended only > for the use of the intended recipient(s), are confidential, and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, retransmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return e-mail, and delete this message and any attachments > from your system. > > > > >> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Antoine, >> >> The NPE error seems to be due to the primary storage with id '5', when >> checking for snapshot revertible or not during list snapshots cmd. Is that >> primary storage removed? Can you check with sql below. >> >> SELECT name, uuid, pool_type, scope, created, removed FROM >> cloud.storage_pool WHERE id = 5 >> >> >> I think, the account/user might not have the permissions to list all >> snapshots. You can check the account / domain details in snapshots table. >> >> >> Regards, >> Suresh >> >> On 21/04/22, 6:50 PM, "Antoine Boucher" <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thank you Suresh, you will find my answers below, >> >> >> >> >> >> >>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Antoine, >>> >>> What is the CloudStack version, where you see this issue. >> >> Latest - 4.16.1 >> >> >> >>> When you noticed this issue, is it while listing snapshots for a particular >>> volume, or all volumes? Try list snapshots from API/cmk as well. >> >> This is a customer VM, the account may have hit its snapshot limit during >> a snapshot. >> >> cmk list snapshots > only lists 26 snapshots out of 62 >> >> cmk list snapshots volumeid= > Works for some volume ids, provides null >> result while the WebUI lists the correct snapshots for others and provides >> null results while show undefined error for one VM mentioned below >> >> >> >>> >>> Also, Check the snapshots details for the volume (with issue) in the db, >>> using the following sql queries. >>> >>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes >>> WHERE name LIKE '<volume_name>' >>> >> >> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM >> cloud.volumes WHERE name LIKE 'ROOT-110'; >> >> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+ >> | id | uuid | state | pool_id | path >> | size | removed | >> >> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+ >> | 111 | NULL | Expunged | 5 | >> da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 | >> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 | >> 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL | >> >> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+ >> >> >> >>> 2. SELECT id, uuid, name, status, path, created, removed, location_type >>> FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>; >> >> mysql> SELECT id, uuid, name, status, path, created, removed, >> location_type FROM cloud.snapshots WHERE volume_id = '138'; >> >> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+ >> | id | uuid | name >> | status | path | created | removed | location_type | >> >> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+ >> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | >> TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 | NULL >> | NULL | >> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | >> TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 | NULL >> | NULL | >> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | >> TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 | NULL >> | NULL | >> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | >> TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 | NULL >> | NULL | >> | 138 | e9639f4a-fa09-4698-849b-572011045993 | >> TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 | NULL >> | NULL | >> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | >> TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 | NULL >> | NULL | >> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | >> TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 | NULL >> | NULL | >> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20 >> | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL | >> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | >> TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 | NULL >> | NULL | >> >> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+ >> >> >> >>> >>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM >>> cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2> >>> >> >> mysql> SELECT volume_id, store_id, store_role, install_path, size, state >> FROM cloud.snapshot_store_ref WHERE snapshot_id = '101'; >> >> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+ >> | volume_id | store_id | store_role | install_path >> | size | state | >> >> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+ >> | 138 | 1 | Image | >> snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 | >> Destroying | >> >> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+ >> 1 row in set (0.00 sec) >> >> mysql> SELECT volume_id, store_id, store_role, install_path, size, state >> FROM cloud.snapshot_store_ref WHERE snapshot_id = '109'; >> >> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+ >> | volume_id | store_id | store_role | install_path >> | size >> | state | >> >> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+ >> | 138 | 5 | Primary | >> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5 >> | 85899345920 | Ready | >> | 138 | 1 | Image | >> snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5 >> | 85899345920 | Ready | >> >> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+ >> 2 rows in set (0.00 sec) >> >> Regards, >> Antoine >> >> >>> Regards, >>> Suresh >>> >>> On 21/04/22, 8:45 AM, "Antoine Boucher" <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Unfortunately I spoke too fast. The list is back but listing the >>> snapshots of the volume with issue still show a blank list with error >>> “undefined”. >>> >>> >>> >>> >>> On Apr 20, 2022, at 23:11, Antoine Boucher <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> I was able to recover from the situation by forcing a snapshot on the >>> volume with issue. >>> >>> On Apr 20, 2022, at 22:00, Antoine Boucher <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> I pin-pointed the volume with the issue by going through all volumes and >>> clicking the “view snapshot" >>> >>> >>> >>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> I’m no longer able to see my list of snapshots (storage > snapshots), >>>> instead I see a small popup window with the word “undefined" on an empty >>>> list snapshot list. >>>> >>>> Has anyone seen this issue? I restart and rebooted the Management server >>>> but no resolution. >>>> >>>> Here are my logs: >>>> >>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet] >>>> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START=== 10.101.254.1 >>>> -- GET listall=true&page=1&pagesize=20&command=listSnapshots&response=json >>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer] >>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from >>>> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account >>>> {"id": 2, "name": "admin", "uuid": >>>> "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls: >>>> 0.0.0.0/0,::/0 >>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer] >>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled >>>> exception executing api command: [Ljava.lang.String;@42fa8f7d >>>> java.lang.NullPointerException >>>> at >>>> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985) >>>> at >>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72) >>>> at >>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69) >>>> at >>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95) >>>> at >>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69) >>>> at >>>> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156) >>>> at >>>> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591) >>>> at >>>> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117) >>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) >>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772) >>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596) >>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321) >>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) >>>> at >>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) >>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131) >>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) >>>> at >>>> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) >>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>> at >>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) >>>> at >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>> at >>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) >>>> at >>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) >>>> at >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) >>>> at org.eclipse.jetty.server.Server.handle(Server.java:516) >>>> at >>>> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400) >>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645) >>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392) >>>> at >>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) >>>> at >>>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) >>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) >>>> at >>>> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) >>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) >>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) >>>> at >>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) >>>> at >>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) >>>> at >>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) >>>> at >>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) >>>> at >>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) >>>> at >>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) >>>> at >>>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) >>>> at java.base/java.lang.Thread.run(Thread.java:829) >>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet] >>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END=== >>>> 10.101.254.1 -- GET >>>> listall=true&page=1&pagesize=20&command=listSnapshots&response=json >>>> >>>> Antoine Boucher >>>> >>>> >>>> Confidentiality Warning: This message and any attachments are intended >>>> only for the use of the intended recipient(s), are confidential, and may >>>> be privileged. If you are not the intended recipient, you are hereby >>>> notified that any review, retransmission, conversion to hard copy, >>>> copying, circulation or other use of this message and any attachments is >>>> strictly prohibited. If you are not the intended recipient, please notify >>>> the sender immediately by return e-mail, and delete this message and any >>>> attachments from your system. >>>>
