HiNo KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
On 10/17/22 15:42, Nicolas Vazquez wrote:
Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?Regards, Nicolas Vazquez *From: *Granwille Strauss <granwi...@namhost.com> *Date: *Monday, 17 October 2022 at 10:04 *To: *users@cloudstack.apache.org <users@cloudstack.apache.org> *Cc: *Nicolas Vazquez <nicolas.vazq...@shapeblue.com> *Subject: *Re: Recurring Snaps Failing Please find attached TXT On 10/17/22 14:54, Nicolas Vazquez wrote: Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796? Regards, Nicolas Vazquez From: Granwille Strauss<granwi...@namhost.com> <mailto:granwi...@namhost.com> Date: Monday, 17 October 2022 at 09:32 To:users@cloudstack.apache.org <users@cloudstack.apache.org> <mailto:users@cloudstack.apache.org> Cc: Nicolas Vazquez<nicolas.vazq...@shapeblue.com> <mailto:nicolas.vazq...@shapeblue.com> Subject: Re: Recurring Snaps Failing Hi This is all agent threw during the snap process nothing else: 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false On 10/17/22 14:29, Nicolas Vazquez wrote: Hi Groete, Can you share the CloudStack agent logs from the KVM host? Regards, Nicolas Vazquez From: Granwille Strauss<granwi...@namhost.com.INVALID> <mailto:granwi...@namhost.com.INVALID><mailto:granwi...@namhost.com.INVALID> <mailto:granwi...@namhost.com.INVALID> Date: Monday, 17 October 2022 at 08:56 To:users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> <mailto:users@cloudstack.apache.org> <users@cloudstack.apache.org> <mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org> <mailto:users@cloudstack.apache.org> Subject: Re: Recurring Snaps Failing Hi I ran a few more tests and can now finally make sense of the failure: 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600 Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped. So I still have no idea where to solve this as all other necessary settings are in place. Any ideas? On 10/17/22 08:51, Granwille Strauss wrote: Hi Guys I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error: root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"} 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"} All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look. Any assistance in this regard will be appreciated. -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com><mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- Regards / Groete [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin e:granwi...@namhost.com<mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>-- Regards / Groete <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin *e:* granwi...@namhost.com <mailto:granwi...@namhost.com> *m:* +264 81 323 1260 <tel:+264813231260> *w:* www.namhost.com <https://www.namhost.com/> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSAThe content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policyPowered by AdSigner <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
-- Regards / Groete <https://www.namhost.com> Granwille Strauss // Senior Systems Admin *e:* granwi...@namhost.com *m:* +264 81 323 1260 <tel:+264813231260> *w:* www.namhost.com <https://www.namhost.com/> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSAThe content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
Powered by AdSigner <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
smime.p7s
Description: S/MIME Cryptographic Signature
