levindecaro opened a new issue, #8640:
URL: https://github.com/apache/cloudstack/issues/8640
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
UI
Snapshot
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.19
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
advanced networking
KVM
SharedMountPoint
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
CentOS 7
##### SUMMARY
<!-- Explain the problem/feature briefly -->
After upgrade 4.19, perform volume snapshot on existing VM will fail
immediately from UI, but the snapshot are actually running on the host, the job
will completed successfully and refresh the status from UI.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
Pick any existing ROOT Volume
Type in Name
Omit the Zone
Async backup: true or false
Quiesce Instance: false
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Snapshot result should be reported after the job complete.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
Error shown from UI, but Job is running on the host and completed
successfully.
~~~
Failed to create snapshot due to an internal error creating snapshot for
volume e8584012-f806-4511-b892-f1cbadcd8b1d
~~~
From management server
```
[root@cloudstack-1 management]# grep 22c506ce management-server.log
2024-02-10 17:05:29,040 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Executing
AsyncJobVO: {id:98011, userId: 9, accountId: 9, instanceType: Snapshot,
instanceId: 3144, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
{"quiescevm":"false","httpmethod":"GET","ctxAccountId":"9","uuid":"5086e6a7-c678-49bd-8d2f-2a4e2add25e4","cmdEventType":"SNAPSHOT.CREATE","asyncBackup":"false","response":"json","ctxUserId":"9","volumeId":"e8584012-f806-4511-b892-f1cbadcd8b1d","name":"podman-builder-backup","ctxStartEventId":"216990","id":"3144","ctxDetails":"{\"interface
com.cloud.storage.Snapshot\":\"5086e6a7-c678-49bd-8d2f-2a4e2add25e4\",\"interface
com.cloud.storage.Volume\":\"e8584012-f806-4511-b892-f1cbadcd8b1d\"}"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 52235250599, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null, removed: null}
2024-02-10 17:05:29,046 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011 ctx-f0ab1c1e) (logid:22c506ce)
Access to Account
[{"accountName":"levin.ng","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}]
granted to Account
[{"accountName":"levin.ng","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}]
by DomainChecker
2024-02-10 17:05:29,075 DEBUG [c.c.a.ApiServlet]
(qtp770947228-990:ctx-0eec9834) (logid:93b255ec) ===START=== 0:0:0:0:0:0:0:1
-- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
2024-02-10 17:05:29,081 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011 ctx-f0ab1c1e) (logid:22c506ce)
Access to org.apache.cloudstack.storage.volume.VolumeObject@6ba903f2 granted to
Account
[{"accountName":"levin.ng","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}]
by DomainChecker
2024-02-10 17:05:29,086 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011 ctx-f0ab1c1e) (logid:22c506ce)
Access to VM instance
{"id":652,"instanceName":"i-9-652-VM","type":"User","uuid":"cc61b965-0a8f-40d6-8986-926566de6299"}
granted to Account
[{"accountName":"levin.ng","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}]
by DomainChecker
2024-02-10 17:05:29,093 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011 ctx-f0ab1c1e) (logid:22c506ce) Sync
job-98012 execution on object VmWorkJobQueue.652
2024-02-10 17:05:29,120 DEBUG [c.c.a.ApiServlet]
(qtp770947228-990:ctx-0eec9834 ctx-ef61473c) (logid:93b255ec) ===END===
0:0:0:0:0:0:0:1 -- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
2024-02-10 17:05:29,929 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012) (logid:22c506ce)
Executing AsyncJobVO: {id:98012, userId: 9, accountId: 9, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAkAAAAAAAAACQAAAAAAAAKMdAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAAMSHNxAH4ACAAAAAAAAAKgcA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 52235250599, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Sat Feb 10 17:05:29 HKT 2024,
removed: null}
2024-02-10 17:05:29,929 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012) (logid:22c506ce) Run VM
work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 652, job origin: 98011
2024-02-10 17:05:29,931 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":672,"policyId":0,"snapshotId":3144,"quiesceVm":false,"asyncBackup":false,"userId":9,"accountId":9,"vmId":652,"handlerName":"VolumeApiServiceImpl"}
2024-02-10 17:05:29,942 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) StorpoolSnapshotStrategy.canHandle:
snapshot=podman-builder-backup, uuid=5086e6a7-c678-49bd-8d2f-2a4e2add25e4,
op=TAKE
2024-02-10 17:05:30,029 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) Taking snapshot of
org.apache.cloudstack.storage.snapshot.SnapshotObject@7d74a2d1
2024-02-10 17:05:30,051 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) Taking snapshot of
org.apache.cloudstack.storage.snapshot.SnapshotObject@7d74a2d1 and encryption
required is false
2024-02-10 17:05:30,052 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2024-02-10 17:05:30,053 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) We are returning the default host to execute commands because
the command is not of Copy type.
2024-02-10 17:05:30,056 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-22:ctx-55908d82 job-98011/job-98012 ctx-1c9a8eaa)
(logid:22c506ce) Seq 49-4468696730258380648: Sending { Cmd , MgmtId:
52235250599, via: 49(compute-15.lab.example.com), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"e8584012-f806-4511-b892-f1cbadcd8b1d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5db4c6c5-cc09-4f99-af42-c2256f76748c","name":"sg1","id":"3","poolType":"SharedMountPoint","host":"localhost","path":"/cloudstack/sg1","port":"0","url":"SharedMountPoint://localhost/cloudstack/sg1/?ROLE=Primary&STOREUUID=5db4c6c5-cc09-4f99-af42-c2256f76748c","isManaged":"false"}},"name":"ROOT-652","size":"(200.00
GB)
214748364800","path":"e8584012-f806-4511-b892-f1cbadcd8b1d","volumeId":"672","vmName":"i-9-652-VM","accountId":"9","format":"QCOW2","provisi
oningType":"THIN","poolId":"3","id":"672","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5db4c6c5-cc09-4f99-af42-c2256f76748c","name":"sg1","id":"3","poolType":"SharedMountPoint","host":"localhost","path":"/cloudstack/sg1","port":"0","url":"SharedMountPoint://localhost/cloudstack/sg1/?ROLE=Primary&STOREUUID=5db4c6c5-cc09-4f99-af42-c2256f76748c","isManaged":"false"}},"vmName":"i-9-652-VM","name":"podman-builder-backup","hypervisorType":"KVM","id":"3144","quiescevm":"false","physicalSize":"0","accountId":"9"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2024-02-10 17:05:32,178 DEBUG [c.c.a.ApiServlet]
(qtp770947228-984:ctx-0c66e32a) (logid:42aae736) ===START=== 0:0:0:0:0:0:0:1
-- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
2024-02-10 17:05:32,206 DEBUG [c.c.a.ApiServlet]
(qtp770947228-984:ctx-0c66e32a ctx-6c86acba) (logid:42aae736) ===END===
0:0:0:0:0:0:0:1 -- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
2024-02-10 17:05:34,141 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-18:ctx-03690884 job-98011 ctx-f0ab1c1e) (logid:22c506ce)
Failed to create snapshot due to an internal error creating snapshot for volume
e8584012-f806-4511-b892-f1cbadcd8b1d
2024-02-10 17:05:34,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Complete async
job-98011, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume
e8584012-f806-4511-b892-f1cbadcd8b1d"}
2024-02-10 17:05:34,151 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Publish async
job-98011 complete on message bus
2024-02-10 17:05:34,151 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Wake up jobs
related to job-98011
2024-02-10 17:05:34,151 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Update db status
for job-98011
2024-02-10 17:05:34,156 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Wake up jobs
joined with job-98011 and disjoin all subjobs created from job- 98011
2024-02-10 17:05:34,160 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Done executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-98011
2024-02-10 17:05:34,160 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-18:ctx-03690884 job-98011) (logid:22c506ce) Remove job-98011
from job monitoring
2024-02-10 17:05:35,249 DEBUG [c.c.a.ApiServlet]
(qtp770947228-990:ctx-8f378aa9) (logid:d9dc87f0) ===START=== 0:0:0:0:0:0:0:1
-- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
2024-02-10 17:05:35,281 DEBUG [c.c.a.ApiServlet]
(qtp770947228-990:ctx-8f378aa9 ctx-3151ec2e) (logid:d9dc87f0) ===END===
0:0:0:0:0:0:0:1 -- GET
jobId=22c506ce-58c4-4cef-a569-017e10057180&command=queryAsyncJobResult&response=json
```
```
no related log from agent
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
