Please find attached TXT On 10/17/22 14:54, Nicolas Vazquez wrote:
Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?Regards, Nicolas Vazquez From: Granwille Strauss<granwi...@namhost.com> Date: Monday, 17 October 2022 at 09:32 To:users@cloudstack.apache.org <users@cloudstack.apache.org> Cc: Nicolas Vazquez<nicolas.vazq...@shapeblue.com> Subject: Re: Recurring Snaps Failing Hi This is all agent threw during the snap process nothing else: 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false On 10/17/22 14:29, Nicolas Vazquez wrote: Hi Groete, Can you share the CloudStack agent logs from the KVM host? Regards, Nicolas Vazquez From: Granwille Strauss<granwi...@namhost.com.INVALID><mailto:granwi...@namhost.com.INVALID> Date: Monday, 17 October 2022 at 08:56 To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <users@cloudstack.apache.org><mailto:users@cloudstack.apache.org> Subject: Re: Recurring Snaps Failing Hi I ran a few more tests and can now finally make sense of the failure: 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600 Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped. So I still have no idea where to solve this as all other necessary settings are in place. Any ideas? On 10/17/22 08:51, Granwille Strauss wrote: Hi Guys I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error: root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"} 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"} All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look. Any assistance in this regard will be appreciated. -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com><mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com><mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260> w:www.namhost.com<https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
-- Regards / Groete <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:* granwi...@namhost.com *m:* +264 81 323 1260 <tel:+264813231260> *w:* www.namhost.com <https://www.namhost.com/> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSAThe content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
Powered by AdSigner <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
root@MANAGEMEN /var/log/cloudstack/management $ grep "job-796"
management-server.log
2022-10-17 02:01:27,881 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:4e30231f) Add job-796 into
job monitoring
2022-10-17 02:01:27,882 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(SnapshotPollTask:ctx-281af8d9) (logid:c6c95cfe) submit async job-796, details:
AsyncJobVO: {id:796, userId: 1, accountId: 2, instanceType: Snapshot,
instanceId: 21, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
{"policyid":"18","ctxUserId":"1","volumeid":"24","ctxStartEventId":"1","id":"21","ctxAccountId":"2"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 2619780474, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null, removed: null}
2022-10-17 02:01:27,882 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Executing
AsyncJobVO: {id:796, userId: 1, accountId: 2, instanceType: Snapshot,
instanceId: 21, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
{"policyid":"18","ctxUserId":"1","volumeid":"24","ctxStartEventId":"1","id":"21","ctxAccountId":"2"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 2619780474, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null, removed: null}
2022-10-17 02:01:27,904 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Sync
job-797 execution on object VmWorkJobQueue.19
2022-10-17 02:01:29,397 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:efd91c0d) Add
job-797 into job monitoring
2022-10-17 02:01:29,400 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Executing
AsyncJobVO: {id:797, userId: 1, accountId: 2, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAEAAAAAAAAAE3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAASc3EAfgAHAAAAAAAAABVzcQB-AAcAAAAAAAAAGA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 2619780474, completeMsid: null, lastUpdated: null, lastPolled:
null, created: Mon Oct 17 02:01:27 SAST 2022, removed: null}
2022-10-17 02:01:29,400 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Run VM
work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 19, job origin: 796
2022-10-17 02:01:29,401 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":24,"policyId":18,"snapshotId":21,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":2,"vmId":19,"handlerName":"VolumeApiServiceImpl"}
2022-10-17 02:01:29,406 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) StorpoolSnapshotStrategy.canHandle:
snapshot=RocketChat_ROOT-19_20221017000127,
uuid=cc3bd9ed-e886-41bf-aec1-491587dce907, op=TAKE
2022-10-17 02:01:29,430 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2022-10-17 02:01:29,430 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) We are returning the default host to execute commands because
the command is not of Copy type.
2022-10-17 02:01:29,432 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Seq 1-2442639847895100340: Sending { Cmd , MgmtId:
2619780474, via: 1(MANAGEMENTHOSTNAME), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"635fd5b5-bf14-4cf4-979c-be8a649d5189","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"81ffca3a-9775-375d-a1c0-9504c0ec3d89","name":"Primary","id":"2","poolType":"NetworkFilesystem","host":"MANAGEMENTIP","path":"/mnt/primary","port":"2049","url":"NetworkFilesystem://MANAGEMENTIP/mnt/primary/?ROLE=Primary&STOREUUID=81ffca3a-9775-375d-a1c0-9504c0ec3d89","isManaged":"false"}},"name":"ROOT-19","size":"(50,00
GB)
53687091200","path":"635fd5b5-bf14-4cf4-979c-be8a649d5189","volumeId":"24","vmName":"i-2-19-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"2","id":"24","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"81ffca3a-9775-375d-a1c0-9504c0ec3d89","name":"Primary","id":"2","poolType":"NetworkFilesystem","host":"MANAGEMENTIP","path":"/mnt/primary","port":"2049","url":"NetworkFilesystem://MANAGEMENTIP/mnt/primary/?ROLE=Primary&STOREUUID=81ffca3a-9775-375d-a1c0-9504c0ec3d89","isManaged":"false"}},"vmName":"i-2-19-VM","name":"RocketChat_ROOT-19_20221017000127","hypervisorType":"KVM","id":"21","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-17 02:03:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-1280d960) (logid:9258af42) Task (job-796) has been pending for 114
seconds
2022-10-17 02:04:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-428e5559) (logid:ac4e0a54) Task (job-796) has been pending for 174
seconds
2022-10-17 02:05:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-4df0a37f) (logid:b0c5d1af) Task (job-796) has been pending for 234
seconds
2022-10-17 02:06:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-b70a39f2) (logid:f79dd307) Task (job-796) has been pending for 294
seconds
2022-10-17 02:07:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-0d3d15a7) (logid:4bc99fce) Task (job-796) has been pending for 354
seconds
2022-10-17 02:08:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-5422f8fd) (logid:d43859ce) Task (job-796) has been pending for 414
seconds
2022-10-17 02:09:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-dea4f954) (logid:a54a3418) Task (job-796) has been pending for 474
seconds
2022-10-17 02:10:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-f21f7561) (logid:bb1e7858) Task (job-796) has been pending for 534
seconds
2022-10-17 02:11:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-a46b75b2) (logid:96bea681) Task (job-796) has been pending for 594
seconds
2022-10-17 02:12:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-aa344072) (logid:013da696) Task (job-796) has been pending for 654
seconds
2022-10-17 02:13:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-9e663977) (logid:fb4360d7) Task (job-796) has been pending for 714
seconds
2022-10-17 02:14:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-a9db5b45) (logid:41d7db04) Task (job-796) has been pending for 774
seconds
2022-10-17 02:15:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-f8684559) (logid:f277976a) Task (job-796) has been pending for 834
seconds
2022-10-17 02:16:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-3b6750f3) (logid:dce2be2e) Task (job-796) has been pending for 894
seconds
2022-10-17 02:17:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-1bf4f123) (logid:775c07ee) Task (job-796) has been pending for 954
seconds
2022-10-17 02:18:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-48cebacd) (logid:ccd2701d) Task (job-796) has been pending for
1014 seconds
2022-10-17 02:19:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-1ec3355d) (logid:e6328cd6) Task (job-796) has been pending for
1074 seconds
2022-10-17 02:20:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-aa4e8209) (logid:83cab643) Task (job-796) has been pending for
1134 seconds
2022-10-17 02:21:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2c7d0cdb) (logid:751a09ca) Task (job-796) has been pending for
1194 seconds
2022-10-17 02:22:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-69464c94) (logid:9007e542) Task (job-796) has been pending for
1254 seconds
2022-10-17 02:23:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-26043fda) (logid:bbf14729) Task (job-796) has been pending for
1314 seconds
2022-10-17 02:24:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-20767843) (logid:12c6de23) Task (job-796) has been pending for
1374 seconds
2022-10-17 02:25:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-084d4849) (logid:7103b377) Task (job-796) has been pending for
1434 seconds
2022-10-17 02:26:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-3772663f) (logid:3c582c22) Task (job-796) has been pending for
1494 seconds
2022-10-17 02:27:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-11afc2d6) (logid:4997439a) Task (job-796) has been pending for
1554 seconds
2022-10-17 02:28:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-df601643) (logid:e30cd8bc) Task (job-796) has been pending for
1614 seconds
2022-10-17 02:29:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-e856ac56) (logid:d94056b9) Task (job-796) has been pending for
1674 seconds
2022-10-17 02:30:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-db823864) (logid:41314333) Task (job-796) has been pending for
1734 seconds
2022-10-17 02:31:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-df6762d5) (logid:94682775) Task (job-796) has been pending for
1794 seconds
2022-10-17 02:31:29,432 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Seq 1-2442639847895100340: Waiting some more time because this
is the current command
2022-10-17 02:32:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-67ab11ef) (logid:e6543ea7) Task (job-796) has been pending for
1854 seconds
2022-10-17 02:33:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2460adb5) (logid:8ea8ec95) Task (job-796) has been pending for
1914 seconds
2022-10-17 02:34:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-07344ca5) (logid:7abefbc3) Task (job-796) has been pending for
1974 seconds
2022-10-17 02:35:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-198d31a8) (logid:6ee74e2e) Task (job-796) has been pending for
2034 seconds
2022-10-17 02:36:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2014ba2e) (logid:090e7423) Task (job-796) has been pending for
2094 seconds
2022-10-17 02:37:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-7fbc6763) (logid:e1f7dce1) Task (job-796) has been pending for
2154 seconds
2022-10-17 02:38:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-f9fc9275) (logid:706a060d) Task (job-796) has been pending for
2214 seconds
2022-10-17 02:39:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-415391a2) (logid:c3b4ab32) Task (job-796) has been pending for
2274 seconds
2022-10-17 02:40:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-4d1bfd94) (logid:6e801dd0) Task (job-796) has been pending for
2334 seconds
2022-10-17 02:41:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-9624667c) (logid:75445060) Task (job-796) has been pending for
2394 seconds
2022-10-17 02:42:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-59f078c8) (logid:59298d6b) Task (job-796) has been pending for
2454 seconds
2022-10-17 02:43:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-e6086b3e) (logid:a982f829) Task (job-796) has been pending for
2514 seconds
2022-10-17 02:44:22,343 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-b9385211) (logid:310bda7f) Task (job-796) has been pending for
2574 seconds
2022-10-17 02:45:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-12e9332d) (logid:1c3be65a) Task (job-796) has been pending for
2634 seconds
2022-10-17 02:46:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2bd549d8) (logid:03a2da39) Task (job-796) has been pending for
2694 seconds
2022-10-17 02:47:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-963ce9cf) (logid:587bc7bd) Task (job-796) has been pending for
2754 seconds
2022-10-17 02:48:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-463b5e1f) (logid:7ec345d0) Task (job-796) has been pending for
2814 seconds
2022-10-17 02:49:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-7864037f) (logid:38692589) Task (job-796) has been pending for
2874 seconds
2022-10-17 02:50:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-39f7bfc0) (logid:dd3637ff) Task (job-796) has been pending for
2934 seconds
2022-10-17 02:51:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-cecb7042) (logid:eb4c6891) Task (job-796) has been pending for
2994 seconds
2022-10-17 02:52:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-f04c153c) (logid:fd30df05) Task (job-796) has been pending for
3054 seconds
2022-10-17 02:53:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-79667d9b) (logid:a8b6a81a) Task (job-796) has been pending for
3114 seconds
2022-10-17 02:54:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2d41f00c) (logid:249b619c) Task (job-796) has been pending for
3174 seconds
2022-10-17 02:55:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-2be2de7a) (logid:03951f90) Task (job-796) has been pending for
3234 seconds
2022-10-17 02:56:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-a800e27c) (logid:1b042a7b) Task (job-796) has been pending for
3294 seconds
2022-10-17 02:57:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-fe4aabdf) (logid:3aaedbd1) Task (job-796) has been pending for
3354 seconds
2022-10-17 02:58:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-217a81d2) (logid:11c9440d) Task (job-796) has been pending for
3414 seconds
2022-10-17 02:59:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-7b24930c) (logid:6cef277c) Task (job-796) has been pending for
3474 seconds
2022-10-17 03:00:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-6e83ddbf) (logid:ecc57f9e) Task (job-796) has been pending for
3534 seconds
2022-10-17 03:01:22,342 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-b9175420) (logid:3e29f725) Task (job-796) has been pending for
3594 seconds
2022-10-17 03:01:29,433 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Seq 1-2442639847895100340: Waiting some more time because this
is the current command
2022-10-17 03:01:29,434 WARN [c.c.a.m.AgentAttache]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Seq 1-2442639847895100340: Timed out on Seq
1-2442639847895100340: { Cmd , MgmtId: 2619780474, via: 1(MANAGEMENTHOSTNAME),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"635fd5b5-bf14-4cf4-979c-be8a649d5189","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"81ffca3a-9775-375d-a1c0-9504c0ec3d89","name":"Primary","id":"2","poolType":"NetworkFilesystem","host":"MANAGEMENTIP","path":"/mnt/primary","port":"2049","url":"NetworkFilesystem://MANAGEMENTIP/mnt/primary/?ROLE=Primary&STOREUUID=81ffca3a-9775-375d-a1c0-9504c0ec3d89","isManaged":"false"}},"name":"ROOT-19","size":"(50,00
GB)
53687091200","path":"635fd5b5-bf14-4cf4-979c-be8a649d5189","volumeId":"24","vmName":"i-2-19-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"2","id":"24","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"81ffca3a-9775-375d-a1c0-9504c0ec3d89","name":"Primary","id":"2","poolType":"NetworkFilesystem","host":"MANAGEMENTIP","path":"/mnt/primary","port":"2049","url":"NetworkFilesystem://MANAGEMENTIP/mnt/primary/?ROLE=Primary&STOREUUID=81ffca3a-9775-375d-a1c0-9504c0ec3d89","isManaged":"false"}},"vmName":"i-2-19-VM","name":"RocketChat_ROOT-19_20221017000127","hypervisorType":"KVM","id":"21","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2022-10-17 03:01:29,434 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Seq 1-2442639847895100340: Cancelling.
2022-10-17 03:01:29,434 DEBUG [o.a.c.s.RemoteHostEndPoint]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Failed to send command, due to Agent:1,
com.cloud.exception.OperationTimedoutException: Commands 2442639847895100340 to
Host 1 timed out after 3600
2022-10-17 03:01:29,434 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Failed to take snapshot: 21
2022-10-17 03:01:29,434 DEBUG [o.a.c.s.s.SnapshotServiceImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) create snapshot RocketChat_ROOT-19_20221017000127 failed:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:1, com.cloud.exception.OperationTimedoutException: Commands
2442639847895100340 to Host 1 timed out after 3600
2022-10-17 03:01:29,444 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Failed to take snapshot:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:1, com.cloud.exception.OperationTimedoutException: Commands
2442639847895100340 to Host 1 timed out after 3600
2022-10-17 03:01:29,449 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Failed to create
snapshotcom.cloud.utils.exception.CloudRuntimeException: Failed to send
command, due to Agent:1, com.cloud.exception.OperationTimedoutException:
Commands 2442639847895100340 to Host 1 timed out after 3600
2022-10-17 03:01:29,450 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Updating resource Type = snapshot count for Account = 2
Operation = decreasing Amount = 1
2022-10-17 03:01:29,453 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Updating resource Type = secondary_storage count for Account =
2 Operation = decreasing Amount = (50,00 GB) 53687091200
2022-10-17 03:01:29,456 ERROR [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Take snapshot: 24 failed
2022-10-17 03:01:29,457 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:1, com.cloud.exception.OperationTimedoutException: Commands
2442639847895100340 to Host 1 timed out after 3600
2022-10-17 03:01:29,457 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797 ctx-0108a34a)
(logid:f9a61b00) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:1, com.cloud.exception.OperationTimedoutException: Commands
2442639847895100340 to Host 1 timed out after 3600
2022-10-17 03:01:29,457 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Done with
run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 19, job
origin: 796
2022-10-17 03:01:29,457 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Unable to
complete AsyncJobVO: {id:797, userId: 1, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAEAAAAAAAAAE3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAASc3EAfgAHAAAAAAAAABVzcQB-AAcAAAAAAAAAGA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 2619780474, completeMsid: null, lastUpdated: null, lastPolled:
null, created: Mon Oct 17 02:01:27 SAST 2022, removed: null}, job origin:796
2022-10-17 03:01:29,460 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Complete
async job-797, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AMRjb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbjogRmFpbGVkIHRvIHNlbmQgY29tbWFuZCwgZHVlIHRvIEFnZW50OjEsIGNvbS5jbG91ZC5leGNlcHRpb24uT3BlcmF0aW9uVGltZWRvdXRFeGNlcHRpb246IENvbW1hbmRzIDI0NDI2Mzk4NDc4OTUxMDAzNDAgdG8gSG9zdCAxIHRpbWVkIG91dCBhZnRlciAzNjAwdXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAuc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAFTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAVMAAptb2R1bGVOYW1lcQB-AAVMAA1tb2R1bGVWZXJzaW9ucQB-AAV4cAEAAAG6dAADYXBwdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLnN0b3JhZ2Uuc25hcHNob3QuRGVmYXVsdFNuYXBzaG90U3RyYXRlZ3l0ABxEZWZhdWx0U25hcHNob3RTdHJhdGVneS5qYXZhdAAMdGFrZVNuYXBzaG90cHBzcQB-AAwBAAAE2HEAfgAOdAAuY29tLmNsb3VkLnN0b3JhZ2Uuc25hcHNob3QuU25hcHNob3RNYW5hZ2VySW1wbHQAGFNuYXBzaG90TWFuYWdlckltcGwuamF2YXEAfgARcHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YXQAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAkxMS4wLjE2LjFzcQB-AAwCAAAAPnBxAH4AFnEAfgAXdAAGaW52b2tlcQB-ABlxAH4AGnNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AHHEAfgAZcQB-ABpzcQB-AAwCAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAccQB-ABlxAH4AGnNxAH4ADAEAAAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdAANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EAfgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9uLmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4AKHEAfgApdAAHcHJvY2VlZHBwc3EAfgAMAQAAAGFxAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-ABxwcHNxAH4ADAEAAAC6cQB-AA5xAH4AKHEAfgApcQB-ACxwcHNxAH4ADAEAAADXcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AHHBwc3EAfgAMAP____9wdAAXY29tLnN1bi5wcm94eS4kUHJveHkyMjFwcQB-ABFwcHNxAH4ADAEAAAqDcQB-AA50ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-ABFwcHNxAH4ADAEAAAzLcQB-AA50ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZVRha2VWb2x1bWVTbmFwc2hvdHBwc3EAfgAMAQAAEONxAH4ADnEAfgA6cQB-ADtxAH4APHBwc3EAfgAMAv____5wcQB-ABZxAH4AF3EAfgAYcQB-ABlxAH4AGnNxAH4ADAIAAAA-cHEAfgAWcQB-ABdxAH4AHHEAfgAZcQB-ABpzcQB-AAwCAAAAK3BxAH4AHnEAfgAfcQB-ABxxAH4AGXEAfgAac3EAfgAMAgAAAjZwcQB-ACFxAH4AInEAfgAccQB-ABlxAH4AGnNxAH4ADAEAAABrcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABDpcQB-AA5xAH4AOnEAfgA7cQB-AEVwcHNxAH4ADAL____-cHEAfgAWcQB-ABdxAH4AGHEAfgAZcQB-ABpzcQB-AAwCAAAAPnBxAH4AFnEAfgAXcQB-ABxxAH4AGXEAfgAac3EAfgAMAgAAACtwcQB-AB5xAH4AH3EAfgAccQB-ABlxAH4AGnNxAH4ADAIAAAI2cHEAfgAhcQB-ACJxAH4AHHEAfgAZcQB-ABpzcQB-AAwBAAABWHEAfgAOcQB-ACRxAH4AJXEAfgAmcHBzcQB-AAwBAAAAxnEAfgAOcQB-AChxAH4AKXEAfgAqcHBzcQB-AAwBAAAAo3EAfgAOcQB-AChxAH4AKXEAfgAscHBzcQB-AAwBAAAAYXEAfgAOcQB-AC5xAH4AL3EAfgAccHBzcQB-AAwBAAAAunEAfgAOcQB-AChxAH4AKXEAfgAscHBzcQB-AAwBAAAA13EAfgAOcQB-ADJxAH4AM3EAfgAccHBzcQB-AAwA_____3B0ABdjb20uc3VuLnByb3h5LiRQcm94eTIyOXBxAH4ARXBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-AGF0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AZHEAfgBhdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AXXEAfgBecHBzcQB-AAwBAAACOHEAfgAOcQB-AFhxAH4AWXEAfgBecHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AYnEAfgAZcQB-ABpzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AF5xAH4AGXEAfgAac3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AGXEAfgAac3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBzcQB-AF5xAH4AGXEAfgAac3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AF5xAH4AGXEAfgAac3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2022-10-17 03:01:29,461 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Publish
async job-797 complete on message bus
2022-10-17 03:01:29,461 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Wake up
jobs related to job-797
2022-10-17 03:01:29,461 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Update db
status for job-797
2022-10-17 03:01:29,462 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Wake up
jobs joined with job-797 and disjoin all subjobs created from job- 797
2022-10-17 03:01:29,466 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Done
executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-797
2022-10-17 03:01:29,467 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-16:ctx-5e0a9da8 job-796/job-797) (logid:f9a61b00) Remove
job-797 from job monitoring
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed
to create snapshot due to an internal error creating snapshot for volume
635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async
job-796, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume
635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,477 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Publish async
job-796 complete on message bus
2022-10-17 03:01:29,477 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Wake up jobs
related to job-796
2022-10-17 03:01:29,477 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Update db status
for job-796
2022-10-17 03:01:29,478 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Wake up jobs joined
with job-796 and disjoin all subjobs created from job- 796
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Done executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-796
2022-10-17 03:01:29,480 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Remove job-796 from
job monitoring
root@MANAGEMEN /var/log/cloudstack/management $
smime.p7s
Description: S/MIME Cryptographic Signature
