Hi Suresh,
Here ids the logs of Cmk delete snapshot id=117 and similar for id=109,127,138
2022-05-16 18:18:49,667 DEBUG [c.c.a.ApiServer]
(qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) CIDRs from
which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account
{"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,689 DEBUG [c.c.a.ApiServlet]
(qtp515715487-147449:ctx-9470efda ctx-7b642401) (logid:dc836839) ===END===
0:0:0:0:0:0:0:1 -- GET
command=deleteSnapshot&id=117&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,689 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:161004ff) Add job-2964 into
job monitoring
2022-05-16 18:18:49,704 DEBUG [c.c.a.ApiServlet]
(qtp515715487-290:ctx-b653031c) (logid:4c211dfd) ===START=== 0:0:0:0:0:0:0:1
-- GET
command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,715 DEBUG [c.c.a.ApiServer] (qtp515715487-290:ctx-b653031c
ctx-61ed5689) (logid:4c211dfd) CIDRs from which account
'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account {"id": 2, "name":
"admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform
API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:49,736 DEBUG [c.c.a.ApiServlet] (qtp515715487-290:ctx-b653031c
ctx-61ed5689) (logid:4c211dfd) ===END=== 0:0:0:0:0:0:0:1 -- GET
command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:49,741 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
java.lang.NullPointerException
at
com.cloud.storage.snapshot.SnapshotManagerImpl.getDataStoreRole(SnapshotManagerImpl.java:1354)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:591)
at jdk.internal.reflect.GeneratedMethodAccessor4669.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy218.deleteSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-16 18:18:49,748 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-204:ctx-1a4cbdc0 job-2964) (logid:e6172481) Remove job-2964
from job monitoring
2022-05-16 18:18:50,692 DEBUG [c.c.a.ApiServlet]
(qtp515715487-147221:ctx-4c1cd6c0) (logid:3652b6cf) ===START===
0:0:0:0:0:0:0:1 -- GET
command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
2022-05-16 18:18:50,706 DEBUG [c.c.a.ApiServer]
(qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) CIDRs from
which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] -- Account
{"id": 2, "name": "admin", "uuid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is
allowed to perform API calls: 0.0.0.0/0,::/0
2022-05-16 18:18:50,723 DEBUG [c.c.a.ApiServlet]
(qtp515715487-147221:ctx-4c1cd6c0 ctx-9cbca8dc) (logid:3652b6cf) ===END===
0:0:0:0:0:0:0:1 -- GET
command=queryAsyncJobResult&jobid=e6172481-968c-4ee4-993e-a9851af91c29&response=json&sessionkey=zLfGQdrCq6kxldkzrnAxYZ8HWQE
My snapshot.backup.to.secondary is configured to ‘true’
The snapshot location 'snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5’
for id=117 no longer exist. All of the concerned snapshots reference
'snapshots/16/111/...' for location but but directory 111 no longer exist. 110
and 112 exists.
Regards,
Antoine
Antoine Boucher
[email protected]
[o] +1-226-505-9734
www.haltondc.com
> On May 13, 2022, at 9:18 AM, Suresh Anaparti <[email protected]>
> wrote:
>
> Hi Antoine,
>
> What is the error in the management server logs while deleting the ‘BackedUp’
> snapshots? Can you share the log if possible.
>
> Are these snapshots backed up to secondary storage as well (and what’s the
> config ‘snapshot.backup.to.secondary’ value when taking snapshot )? If yes,
> can you find the snapshot files at the install_path there? You can check the
> install_path details in cloud.snapshot_store_ref table with volume_id: 155
> and store_role: Image.
>
>
> Regards,
> Suresh
>
>
>
>
> From: Antoine Boucher <[email protected] <mailto:[email protected]>>
> Reply to: "[email protected] <mailto:[email protected]>"
> <[email protected] <mailto:[email protected]>>
> Date: Wednesday, 11 May 2022 at 9:15 PM
> To: users <[email protected]>
> Subject: Re: NPE while listing Snapshots
>
> Too fast id=146 also failed
>
> {
> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
> "completed": "2022-05-11T11:43:01-0400",
> "created": "2022-05-11T11:38:00-0400",
> "jobid": "0bd30e5c-2e61-4988-8152-36835b791769",
> "jobinstanceid": "875d0298-6ed7-43fc-a468-da6042388ac0",
> "jobinstancetype": "Snapshot",
> "jobprocstatus": 0,
> "jobresult": {
> "errorcode": 530,
> "errortext": "Failed to delete
> snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to remove
> snapshot "
> },
> "jobresultcode": 530,
> "jobresulttype": "object",
> "jobstatus": 2,
> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
> }
> 🙈 Error: async API failed for job 0bd30e5c-2e61-4988-8152-36835b791769
>
>
>
>
>
> Antoine Boucher
> [email protected] <mailto:[email protected]>
> [o] +1-226-505-9734
> www.haltondc.com <http://www.haltondc.com/>
>
> “Data security made simple and affordable”
>
>
>
>
>
>
>
> Confidentiality Warning: This message and any attachments are intended only
> for the use of the intended recipient(s), are confidential, and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, retransmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return e-mail, and delete this message and any attachments
> from your system.
>
>
>
>
>> On May 11, 2022, at 11:41 AM, Antoine Boucher <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hello Suresh,
>>
>> That primary storage was part of the local storage of a host that was
>> removed.
>>
>> SELECT name, uuid, pool_type, scope, created, removed FROM
>> cloud.storage_pool WHERE id = 5
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | name | uuid | pool_type | scope | created | removed
>> |
>> +----------------+------+------------+-------+---------------------+---------------------+
>> | cs-kvm03-local | NULL | Filesystem | HOST | 2022-02-12 15:01:45 |
>> 2022-04-19 02:11:51 |
>> +----------------+------+------------+-------+---------------------+——————————+
>>
>>
>>
>>
>> SELECT volume_id, store_id, snapshot_id, store_role, install_path, size,
>> state FROM cloud.snapshot_store_ref WHERE store_id = '5';
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | volume_id | store_id | snapshot_id | store_role | install_path
>> |
>> size | state |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>> | 155 | 5 | 109 | Primary |
>> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
>> | 85899345920 | Ready |
>> | 155 | 5 | 117 | Primary |
>> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/a2c4fbbb-12b2-4c43-9647-375f6e334a46
>> | 85899345920 | Ready |
>> | 155 | 5 | 127 | Primary |
>> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/794d6dff-ba45-4a56-8377-1e48034bf014
>> | 85899345920 | Ready |
>> | 155 | 5 | 138 | Primary |
>> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/38648d51-bb19-49a2-b01f-12c4e6ed4529
>> | 85899345920 | Ready |
>> +-----------+----------+-------------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>
>>
>>
>>
>> All 155 snapshots have the same account_id and domain_id.
>>
>>
>>
>>
>> SELECT id, uuid, name, status, path, created, removed, location_type FROM
>> cloud.snapshots WHERE volume_id = '155';
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | id | uuid | name
>> | status | path | created | removed |
>> location_type |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 | TS01_ROOT-110_20220414054855
>> | Destroyed | NULL | 2022-04-14 05:48:55 | NULL | NULL
>> |
>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 | TS01_ROOT-110_20220415054855
>> | BackedUp | NULL | 2022-04-15 05:48:55 | NULL | NULL
>> |
>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a | TS01_ROOT-110_20220416054855
>> | BackedUp | NULL | 2022-04-16 05:48:55 | NULL | NULL
>> |
>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df | TS01_ROOT-110_20220417054855
>> | BackedUp | NULL | 2022-04-17 05:48:55 | NULL | NULL
>> |
>> | 138 | e9639f4a-fa09-4698-849b-572011045993 | TS01_ROOT-110_20220418054855
>> | BackedUp | NULL | 2022-04-18 05:48:55 | NULL | NULL
>> |
>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 | TS01_ROOT-110_20220419054747
>> | BackedUp | NULL | 2022-04-19 05:47:47 | NULL | NULL
>> |
>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b | TS01_ROOT-110_20220420054748
>> | Destroyed | NULL | 2022-04-20 05:47:48 | NULL | NULL
>> |
>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20
>> | Destroyed | NULL | 2022-04-21 01:50:16 | NULL | NULL
>> |
>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 | TS01_ROOT-110_20220421054817
>> | Destroyed | NULL | 2022-04-21 05:48:17 | NULL | NULL
>> |
>> | 172 | 6e03ea90-e8b0-4405-aed6-7bd17a40de57 | TS01_ROOT-110_20220422054933
>> | Destroyed | NULL | 2022-04-22 05:49:33 | NULL | NULL
>> |
>> | 181 | 9059a26a-0e4e-4d46-95f2-b0a9236ba90a | TS01_ROOT-110_20220423054933
>> | Destroyed | NULL | 2022-04-23 05:49:33 | NULL | NULL
>> |
>> | 193 | c9828cbc-f282-4dfe-b4cf-9df792c522c2 | TS01_ROOT-110_20220424054934
>> | BackedUp | NULL | 2022-04-24 05:49:34 | NULL | NULL
>> |
>> | 207 | 2e5dfd48-c4de-45ff-bbe1-4cf0d90bb698 | TS01_ROOT-110_20220425054934
>> | Destroyed | NULL | 2022-04-25 05:49:34 | NULL | NULL
>> |
>> | 217 | 99d9d9ff-44ae-41a9-9761-f722b370c01c | TS01_ROOT-110_20220426054934
>> | Destroyed | NULL | 2022-04-26 05:49:34 | NULL | NULL
>> |
>> | 227 | 1df14a7b-ba56-4d26-aeb4-c79d4de978fc | TS01_ROOT-110_20220427054934
>> | BackedUp | NULL | 2022-04-27 05:49:34 | NULL | NULL
>> |
>> | 237 | 657479e0-3587-4a72-a8c8-bd7dff54a5e4 | TS01_ROOT-110_20220428054934
>> | BackedUp | NULL | 2022-04-28 05:49:34 | NULL | NULL
>> |
>> | 248 | 2ef0b539-be01-4b0c-9fa5-5fc9c60a6a96 | TS01_ROOT-110_20220429054935
>> | BackedUp | NULL | 2022-04-29 05:49:35 | NULL | NULL
>> |
>> | 257 | 8ab75270-8e3d-471e-8e4a-8d5f3cbc6bc1 | TS01_ROOT-110_20220430054935
>> | BackedUp | NULL | 2022-04-30 05:49:35 | NULL | NULL
>> |
>> | 259 | a1e77bec-8fc7-457f-b0a5-961c339487f7 | TS01_ROOT-110_20220430060935
>> | Destroyed | NULL | 2022-04-30 06:09:35 | 2022-05-09 20:02:38 | NULL
>> |
>> | 313 | 8ad0c04a-1d74-4d51-8223-a6aa3a82d62c | TS01_ROOT-110_20220509015742
>> | BackedUp | NULL | 2022-05-09 01:57:42 | NULL | NULL
>> |
>> | 327 | 0f5e9366-747f-4a34-bec4-1fd31c461ac3 | TS01_ROOT-110_20220509170242
>> | BackedUp | NULL | 2022-05-09 17:02:42 | NULL | NULL
>> |
>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------------------+---------------+
>>
>> How would you suggest I resolve the issue moving forward?
>>
>> I tried the following with error…
>>
>> (nimbus) 🐱 > delete snapshot id=109
>> {
>> "accountid": "2cee75f9-8bc4-11ec-9c43-001e67fd4838",
>> "cmd": "org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd",
>> "completed": "2022-05-11T11:32:08-0400",
>> "created": "2022-05-11T11:32:08-0400",
>> "jobid": "29661b2b-46c5-42bd-9974-d1a226f3fe68",
>> "jobinstanceid": "77d84bf6-f601-44a6-a239-821f735a7f48",
>> "jobinstancetype": "Snapshot",
>> "jobprocstatus": 0,
>> "jobresult": {
>> "errorcode": 530,
>> "errortext": "Command failed due to Internal Server Error"
>> },
>> "jobresultcode": 530,
>> "jobresulttype": "object",
>> "jobstatus": 2,
>> "userid": "2cefb708-8bc4-11ec-9c43-001e67fd4838"
>> }
>> 🙈 Error: async API failed for job 29661b2b-46c5-42bd-9974-d1a226f3fe68
>>
>>
>> Deleting some table rows in the right sequence?
>>
>> Same issue with id=117 to id=138, but I was able to delete id=146
>>
>> Regards,
>> Antoine
>>
>>
>>
>>
>>> On Apr 27, 2022, at 9:00 AM, Suresh Anaparti <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hi Antoine,
>>>
>>> The NPE error seems to be due to the primary storage with id '5', when
>>> checking for snapshot revertible or not during list snapshots cmd. Is that
>>> primary storage removed? Can you check with sql below.
>>>
>>> SELECT name, uuid, pool_type, scope, created, removed FROM
>>> cloud.storage_pool WHERE id = 5
>>>
>>>
>>> I think, the account/user might not have the permissions to list all
>>> snapshots. You can check the account / domain details in snapshots table.
>>>
>>>
>>> Regards,
>>> Suresh
>>>
>>> On 21/04/22, 6:50 PM, "Antoine Boucher" <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Thank you Suresh, you will find my answers below,
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Apr 21, 2022, at 1:02 AM, Suresh Anaparti
>>>> <[email protected] <mailto:[email protected]>>
>>>> wrote:
>>>>
>>>> Hi Antoine,
>>>>
>>>> What is the CloudStack version, where you see this issue.
>>>
>>> Latest - 4.16.1
>>>
>>>
>>>
>>>> When you noticed this issue, is it while listing snapshots for a
>>>> particular volume, or all volumes? Try list snapshots from API/cmk as well.
>>>
>>> This is a customer VM, the account may have hit its snapshot limit
>>> during a snapshot.
>>>
>>> cmk list snapshots > only lists 26 snapshots out of 62
>>>
>>> cmk list snapshots volumeid= > Works for some volume ids, provides null
>>> result while the WebUI lists the correct snapshots for others and provides
>>> null results while show undefined error for one VM mentioned below
>>>
>>>
>>>
>>>>
>>>> Also, Check the snapshots details for the volume (with issue) in the db,
>>>> using the following sql queries.
>>>>
>>>> 1. SELECT id, uuid, state, pool_id, path, size, removed FROM cloud.volumes
>>>> WHERE name LIKE '<volume_name>'
>>>>
>>>
>>> mysql> SELECT id, uuid, state, pool_id, path, size, removed FROM
>>> cloud.volumes WHERE name LIKE 'ROOT-110';
>>>
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | id | uuid | state | pool_id | path
>>> | size | removed |
>>>
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>> | 111 | NULL | Expunged | 5 |
>>> da585fd4-b10f-480b-9287-45ecb28b6355 | 85899345920 | 2022-04-19 01:29:08 |
>>> | 138 | da585fd4-b10f-480b-9287-45ecb28b6355 | Ready | 6 |
>>> 04bd5d13-72ca-4240-93ce-ed1b015d3437 | 85899345920 | NULL |
>>>
>>> +-----+--------------------------------------+----------+---------+--------------------------------------+-------------+---------------------+
>>>
>>>
>>>
>>>> 2. SELECT id, uuid, name, status, path, created, removed, location_type
>>>> FROM cloud.snapshots WHERE volume_id = <volume_id_from_step_1>;
>>>
>>> mysql> SELECT id, uuid, name, status, path, created, removed,
>>> location_type FROM cloud.snapshots WHERE volume_id = '138';
>>>
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | id | uuid | name
>>> | status | path | created | removed | location_type |
>>>
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>> | 101 | 13870bbd-d7d7-4664-a22f-f09c80a56d41 |
>>> TS01_ROOT-110_20220414054855 | Destroyed | NULL | 2022-04-14 05:48:55 |
>>> NULL | NULL |
>>> | 109 | 77d84bf6-f601-44a6-a239-821f735a7f48 |
>>> TS01_ROOT-110_20220415054855 | BackedUp | NULL | 2022-04-15 05:48:55 |
>>> NULL | NULL |
>>> | 117 | 7db02f43-a9d8-4c73-905a-1524d90f570a |
>>> TS01_ROOT-110_20220416054855 | BackedUp | NULL | 2022-04-16 05:48:55 |
>>> NULL | NULL |
>>> | 127 | 47288ab9-2da2-40f2-98e3-9ced3f8e87df |
>>> TS01_ROOT-110_20220417054855 | BackedUp | NULL | 2022-04-17 05:48:55 |
>>> NULL | NULL |
>>> | 138 | e9639f4a-fa09-4698-849b-572011045993 |
>>> TS01_ROOT-110_20220418054855 | BackedUp | NULL | 2022-04-18 05:48:55 |
>>> NULL | NULL |
>>> | 146 | 875d0298-6ed7-43fc-a468-da6042388ac0 |
>>> TS01_ROOT-110_20220419054747 | BackedUp | NULL | 2022-04-19 05:47:47 |
>>> NULL | NULL |
>>> | 154 | 0122c9b3-a9f8-473f-a8d7-6e69ba77265b |
>>> TS01_ROOT-110_20220420054748 | BackedUp | NULL | 2022-04-20 05:47:48 |
>>> NULL | NULL |
>>> | 157 | fe2248c6-aa80-4e9f-b539-7f495f1ff39f | Manual - Apr 20
>>> | BackedUp | NULL | 2022-04-21 01:50:16 | NULL | NULL |
>>> | 163 | 32923ad7-cbe3-4708-95d8-23eca921d711 |
>>> TS01_ROOT-110_20220421054817 | BackedUp | NULL | 2022-04-21 05:48:17 |
>>> NULL | NULL |
>>>
>>> +-----+--------------------------------------+------------------------------+-----------+------+---------------------+---------+---------------+
>>>
>>>
>>>
>>>>
>>>> 3. SELECT volume_id, store_id, store_role, install_path, size, state FROM
>>>> cloud.snapshot_store_ref WHERE snapshot_id = <snapshot_id_from_step_2>
>>>>
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state
>>> FROM cloud.snapshot_store_ref WHERE snapshot_id = '101';
>>>
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | volume_id | store_id | store_role | install_path
>>> | size | state |
>>>
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> | 138 | 1 | Image |
>>> snapshots/16/111/a6b1bead-ca9a-464b-82c1-d631611ad790 | 85899345920 |
>>> Destroying |
>>>
>>> +-----------+----------+------------+-------------------------------------------------------+-------------+------------+
>>> 1 row in set (0.00 sec)
>>>
>>> mysql> SELECT volume_id, store_id, store_role, install_path, size, state
>>> FROM cloud.snapshot_store_ref WHERE snapshot_id = '109';
>>>
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | volume_id | store_id | store_role | install_path
>>> | size
>>> | state |
>>>
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> | 138 | 5 | Primary |
>>> /var/lib/libvirt/images/da585fd4-b10f-480b-9287-45ecb28b6355/66210b31-7445-4586-89ea-52a37f5b1ab5
>>> | 85899345920 | Ready |
>>> | 138 | 1 | Image |
>>> snapshots/16/111/66210b31-7445-4586-89ea-52a37f5b1ab5
>>> | 85899345920 | Ready |
>>>
>>> +-----------+----------+------------+---------------------------------------------------------------------------------------------------+-------------+-------+
>>> 2 rows in set (0.00 sec)
>>>
>>> Regards,
>>> Antoine
>>>
>>>
>>>> Regards,
>>>> Suresh
>>>>
>>>> On 21/04/22, 8:45 AM, "Antoine Boucher" <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Unfortunately I spoke too fast. The list is back but listing the
>>>> snapshots of the volume with issue still show a blank list with error
>>>> “undefined”.
>>>>
>>>>
>>>>
>>>>
>>>> On Apr 20, 2022, at 23:11, Antoine Boucher <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> I was able to recover from the situation by forcing a snapshot on the
>>>> volume with issue.
>>>>
>>>> On Apr 20, 2022, at 22:00, Antoine Boucher <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> I pin-pointed the volume with the issue by going through all volumes and
>>>> clicking the “view snapshot"
>>>>
>>>>
>>>>
>>>>> On Apr 20, 2022, at 9:35 PM, Antoine Boucher <[email protected]
>>>>> <mailto:[email protected]>> wrote:
>>>>>
>>>>> I’m no longer able to see my list of snapshots (storage > snapshots),
>>>>> instead I see a small popup window with the word “undefined" on an empty
>>>>> list snapshot list.
>>>>>
>>>>> Has anyone seen this issue? I restart and rebooted the Management server
>>>>> but no resolution.
>>>>>
>>>>> Here are my logs:
>>>>>
>>>>> 2022-04-20 21:17:02,428 DEBUG [c.c.a.ApiServlet]
>>>>> (qtp515715487-292:ctx-a220f915) (logid:624389ae) ===START===
>>>>> 10.101.254.1 -- GET
>>>>> listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>> 2022-04-20 21:17:02,439 DEBUG [c.c.a.ApiServer]
>>>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) CIDRs from
>>>>> which account 'Acct[2cee75f9-8bc4-11ec-9c43-001e67fd4838-admin] --
>>>>> Account {"id": 2, "name": "admin", "uuid":
>>>>> "2cee75f9-8bc4-11ec-9c43-001e67fd4838"}' is allowed to perform API calls:
>>>>> 0.0.0.0/0,::/0
>>>>> 2022-04-20 21:17:02,675 ERROR [c.c.a.ApiServer]
>>>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) unhandled
>>>>> exception executing api command: [Ljava.lang.String;@42fa8f7d
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.cloudstack.storage.snapshot.StorageSystemSnapshotStrategy.canHandle(StorageSystemSnapshotStrategy.java:985)
>>>>> at
>>>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:72)
>>>>> at
>>>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl$3.canHandle(StorageStrategyFactoryImpl.java:69)
>>>>> at
>>>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.bestMatch(StorageStrategyFactoryImpl.java:95)
>>>>> at
>>>>> org.apache.cloudstack.storage.helper.StorageStrategyFactoryImpl.getSnapshotStrategy(StorageStrategyFactoryImpl.java:69)
>>>>> at
>>>>> org.apache.cloudstack.storage.snapshot.SnapshotObject.isRevertable(SnapshotObject.java:156)
>>>>> at
>>>>> com.cloud.api.ApiResponseHelper.createSnapshotResponse(ApiResponseHelper.java:591)
>>>>> at
>>>>> org.apache.cloudstack.api.command.user.snapshot.ListSnapshotsCmd.execute(ListSnapshotsCmd.java:117)
>>>>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
>>>>> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
>>>>> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
>>>>> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
>>>>> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
>>>>> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
>>>>> at
>>>>> org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
>>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
>>>>> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>>>> at
>>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
>>>>> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
>>>>> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
>>>>> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
>>>>> at
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:516)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
>>>>> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
>>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
>>>>> at
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
>>>>> at
>>>>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at
>>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
>>>>> at
>>>>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
>>>>> at
>>>>> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
>>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
>>>>> at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
>>>>> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>> 2022-04-20 21:17:02,677 DEBUG [c.c.a.ApiServlet]
>>>>> (qtp515715487-292:ctx-a220f915 ctx-c0fe9f91) (logid:624389ae) ===END===
>>>>> 10.101.254.1 -- GET
>>>>> listall=true&page=1&pagesize=20&command=listSnapshots&response=json
>>>>>
>>>>> Antoine Boucher
>>>>>
>>>>>
>>>>> Confidentiality Warning: This message and any attachments are intended
>>>>> only for the use of the intended recipient(s), are confidential, and may
>>>>> be privileged. If you are not the intended recipient, you are hereby
>>>>> notified that any review, retransmission, conversion to hard copy,
>>>>> copying, circulation or other use of this message and any attachments is
>>>>> strictly prohibited. If you are not the intended recipient, please notify
>>>>> the sender immediately by return e-mail, and delete this message and any
>>>>> attachments from your system.
>>>>>
