Hello, I am right now working on a fix for this. Please take a look at issue: https://github.com/apache/cloudstack/issues/4498
Until then, the recommendation for those who want to take RBD snapshots is to set global settings *snapshot.backup.to.secondary* to *false.* Regards, Gabriel. Em seg., 7 de dez. de 2020 às 12:23, Piotr Pisz <pi...@piszki.pl> escreveu: > Hello, > > > > Andrija, Rakesh, can you elaborate and explain this: “it's a known issue > and that's why the "path" is missing” ? > > I have this bug all the time, snapshots are done correctly on Ceph / RBD > but we can't make new templates from them. > > Each attempt to copy a volume to Secondary Storage ends with this message > (XML error: missing storage pool source). > > Let's save ourselves by loading templates from external URLs, no problem > here. > > > > Best regards, > > Piotr > > > > > > > > From: Andrija Panic <andrija.pa...@gmail.com> > Sent: Wednesday, November 18, 2020 1:08 AM > To: users <users@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl> > Subject: Re: snapshot is not working > > > > that actually makes it broken with the default behaviour of copying the > snapshots to Secondary Storage (NOT copying it to Sec Stor is > supported/implemented ONLY for KVM/Ceph combo) > > > > You might want to help, by raising a bug in the GitHub... it's supposed to > work > > > > Best. > > > > On Mon, 9 Nov 2020 at 18:51, Piotr Pisz <pi...@piszki.pl <mailto: > pi...@piszki.pl> > wrote: > > Thank You Rakesh! > > After set snapshot.to.secondary to FALSE snapshot on RBD work 😊 > > Best regards, > Piotr > > > > -----Original Message----- > From: Rakesh v <www.rakeshv....@gmail.com <mailto: > www.rakeshv....@gmail.com> > > Sent: Monday, November 9, 2020 5:34 PM > To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> > Subject: Re: snapshot is not working > > Is the global setting which contains the string "snaphot.to.secondary" is > false? If so it's a known issue and that's why the "path" is missing > > Sent from my iPhone > > > On 09-Nov-2020, at 1:22 PM, Piotr Pisz <pi...@piszki.pl <mailto: > pi...@piszki.pl> > wrote: > > > > Hello, > > > > > > > > I need help, snapshots are not working (see below), this is a new CS > > installation on Ceph RBD. Besides, everything works fine, could you > > tell us what might be wrong? > > > > > > > > Take snapshot: 544 failed > > > > com.cloud.utils.exception.CloudRuntimeException: > > com.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > > > > > ??? > > > > > > > > Best regards, > > > > Piotr > > > > > > > > > > > > 2020-11-09 12:56:46,525 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:836f528d) Add > > job-1894 into job monitoring > > > > 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d) submit > > async job-1894, details: AsyncJobVO {id:1894, userId: 2, accountId: 2, > > instanceType: Snapshot, instanceId: 13, cmd: > > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > > cmdInfo: > > {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f > > 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a > > syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945 > > a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427 > > 0","id":"13","ctxDetails":"{\"interface > > com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\" > > interface > > com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"} > > ","_":"1604922798590"}, > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > result: null, initMsid: 176206389407942, completeMsid: null, > lastUpdated: > > null, lastPolled: null, created: null, removed: null} > > > > 2020-11-09 12:56:46,528 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Executing > > AsyncJobVO {id:1894, userId: 2, accountId: 2, instanceType: Snapshot, > > instanceId: 13, cmd: > > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > > cmdInfo: > > {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"82f > > 3faf3-e09c-4eaf-886c-d81bad998b64","cmdEventType":"SNAPSHOT.CREATE","a > > syncBackup":"false","response":"json","ctxUserId":"2","volumeId":"9945 > > a23b-a376-4f85-b378-9f7da8017dac","name":"test","ctxStartEventId":"427 > > 0","id":"13","ctxDetails":"{\"interface > > com.cloud.storage.Volume\":\"9945a23b-a376-4f85-b378-9f7da8017dac\",\" > > interface > > com.cloud.storage.Snapshot\":\"82f3faf3-e09c-4eaf-886c-d81bad998b64\"} > > ","_":"1604922798590"}, > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > result: null, initMsid: 176206389407942, completeMsid: null, > lastUpdated: > > null, lastPolled: null, created: null, removed: null} > > > > 2020-11-09 12:56:46,528 DEBUG [c.c.a.ApiServlet] > > (qtp1497845528-1209:ctx-a9fac85e ctx-7d1c63e1) (logid:a5ddc58d) > > ===END=== > > 10.89.11.31 -- GET > > command=createSnapshot&response=json&volumeId=9945a23b-a376-4f85-b378- > > 9f7da8017dac&quiescevm=false&asyncBackup=false&name=test&_=16049227985 > > 90 > > > > 2020-11-09 12:56:46,549 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (API-Job-Executor-35:ctx-326b32af job-1894 ctx-f8b08221) > > (logid:db479fbc) Sync job-1897 execution on object VmWorkJobQueue.183 > > > > 2020-11-09 12:56:46,850 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-13:null) (logid:) Seq 1-8981303556883622697: > > Routing from 176206389408022 > > > > 2020-11-09 12:56:46,851 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-6:null) (logid:) Seq 1-8981303556883622697: > > MgmtId > > 176206389408022: Resp: Routing to peer > > > > 2020-11-09 12:56:47,086 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-3:null) (logid:) Seq 43-2584503236157253953: > > Routing from 176206389408022 > > > > 2020-11-09 12:56:47,126 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-14:null) (logid:) Seq 43-2584503236157253953: > > MgmtId > > 176206389408022: Resp: Routing to peer > > > > 2020-11-09 12:56:47,131 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-15:null) (logid:) Seq 46-3634967849241411607: > > Routing from 176206389408022 > > > > 2020-11-09 12:56:47,133 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-10:null) (logid:) Seq 46-3634967849241411607: > > MgmtId > > 176206389408022: Resp: Routing to peer > > > > 2020-11-09 12:56:47,181 DEBUG [c.c.s.StatsCollector] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) StorageCollector is > > running... > > > > 2020-11-09 12:56:47,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) > getCommandHostDelegation: > > class com.cloud.agent.api.GetStorageStatsCommand > > > > 2020-11-09 12:56:47,184 DEBUG [c.c.h.XenServerGuru] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the > > default host to execute commands because the command is not of Copy type. > > > > 2020-11-09 12:56:47,230 DEBUG [c.c.a.t.Request] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq > > 58-4654188739910703033: Received: { Ans: , MgmtId: 176206389407942, > via: > > 58(s-10-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } } > > > > 2020-11-09 12:56:47,231 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) > getCommandHostDelegation: > > class com.cloud.agent.api.GetStorageStatsCommand > > > > 2020-11-09 12:56:47,231 DEBUG [c.c.h.XenServerGuru] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) We are returning the > > default host to execute commands because the command is not of Copy type. > > > > 2020-11-09 12:56:47,232 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (StatsCollector-6:ctx-100fc604) (logid:fde56d09) Seq > > 37-1383449510532885592: Forwarding null to 176206389407982 > > > > 2020-11-09 12:56:47,406 DEBUG [c.c.a.ApiServlet] > > (qtp1497845528-1280:ctx-b087af82) (logid:3a8983a6) ===START=== > > 10.89.11.31 -- GET > > command=listInfrastructure&response=json&_=1604922838197 > > > > 2020-11-09 12:56:47,410 DEBUG [c.c.a.ApiServer] > > (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6) CIDRs > > from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]' > > is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0> > > > > 2020-11-09 12:56:47,416 DEBUG [c.c.a.ApiServlet] > > (qtp1497845528-1280:ctx-b087af82 ctx-052d26fb) (logid:3a8983a6) > > ===END=== > > 10.89.11.31 -- GET > > command=listInfrastructure&response=json&_=1604922838197 > > > > 2020-11-09 12:56:47,634 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Execute > > sync-queue > > item: SyncQueueItemVO {id:841, queueId: 601, contentType: AsyncJob, > > contentId: 1897, lastProcessMsid: 176206389407942, lastprocessNumber: > > 5, > > lastProcessTime: Mon Nov 09 12:56:47 CET 2020, created: Mon Nov 09 > > 12:56:46 CET 2020} > > > > 2020-11-09 12:56:47,635 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (AsyncJobMgr-Heartbeat-1:ctx-ccb57103) (logid:56d3b696) Schedule > > queued > > job-1897 > > > > 2020-11-09 12:56:47,637 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:1f589139) > > Add > > job-1897 into job monitoring > > > > 2020-11-09 12:56:47,640 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Executing AsyncJobVO {id:1897, userId: 2, accountId: 2, instanceType: > > null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, > > cmdInfo: > > rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c > > 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v > > Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam > > F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj > > b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA > > R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA > > AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy > > 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC > > AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA, > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > result: null, initMsid: 176206389407942, completeMsid: null, > lastUpdated: > > null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed: > > null} > > > > 2020-11-09 12:56:47,640 DEBUG [c.c.v.VmWorkJobDispatcher] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 183, job > origin: > > 1894 > > > > 2020-11-09 12:56:47,642 DEBUG [c.c.v.VmWorkJobHandlerProxy] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Execute VM work job: > > com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":544,"policyId":0,"sna > > pshotId":13,"quiesceVm":false,"asyncBackup":false,"userId":2,"accountI > > d":2,"vmId":183,"handlerName":"VolumeApiServiceImpl"} > > > > 2020-11-09 12:56:47,675 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) getCommandHostDelegation: class > > org.apache.cloudstack.storage.command.CreateObjectCommand > > > > 2020-11-09 12:56:47,675 DEBUG [c.c.h.XenServerGuru] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) We are returning the default host to execute commands > > because the command is not of Copy type. > > > > 2020-11-09 12:56:47,676 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Seq 40-1407656358530002013: Forwarding Seq > > 40-1407656358530002013: { Cmd , MgmtId: 176206389407942, via: > > 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags: 100011, > > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data": > > {"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid" > > :"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT","dataStore > > ":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"22d1 > > 051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","host":"rbd. > > cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cenagis.local > > /rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245f5477","is > > Managed":false}},"name":"ROOT-183","size":107374182400,"path":"9945a23 > > b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2-183-VM","a > > ccountId":2,"format":"RAW","provisioningType":"THIN","id":544,"deviceI > > d":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directDownload":f > > alse},"dataStore":{"org.apache.cloudstack.storage.to < > http://apache.cloudstack.storage.to> .PrimaryDataStoreT > > O":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"R > > BD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://r > > bd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f > > 909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name":"test"," > > hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSize":0}},"wa > > it":0}}] > > } to 176206389407982 > > > > 2020-11-09 12:56:47,976 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-1:null) (logid:) Seq 1-8603282663161211720: > > Routing from 176206389407982 > > > > 2020-11-09 12:56:47,982 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (AgentManager-Handler-4:null) (logid:) Seq 1-8981303556883622698: > > Routing from 176206389408022 > > > > 2020-11-09 12:56:48,043 DEBUG [c.c.c.ConsoleProxyManagerImpl] > > (consoleproxy-1:ctx-79a33e59) (logid:4c5fdc64) Zone 1 is ready to > > launch console proxy > > > > 2020-11-09 12:56:48,150 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > > (secstorage-1:ctx-c8a78ea1) (logid:473dea1d) Zone 1 is ready to launch > > secondary storage VM > > > > 2020-11-09 12:56:48,190 DEBUG [c.c.a.t.Request] > > (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq > > 1-8552617167353293650: Received: { Ans: , MgmtId: 176206389407942, via: > > 1(srv-oz-r01-n01.cenagis.local), Ver: v1, Flags: 10, { > > GetVmStatsAnswer } } > > > > 2020-11-09 12:56:48,190 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-5:null) (logid:) Seq 1-8603282663161211720: > > MgmtId > > 176206389407982: Resp: Routing to peer > > > > 2020-11-09 12:56:48,198 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > (AgentManager-Handler-11:null) (logid:) Seq 1-8981303556883622698: > > MgmtId > > 176206389408022: Resp: Routing to peer > > > > 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request] > > (AgentManager-Handler-12:null) (logid:) Seq 40-1407656358530002013: > > Processing: { Ans: , MgmtId: 176206389407942, via: 40, Ver: v1, Flags: > > 10, > > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{ > > "org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"rbd/9945a > > 23b-a376-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19", > > "id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}] > > } > > > > 2020-11-09 12:56:48,212 DEBUG [c.c.a.t.Request] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Seq 40-1407656358530002013: Received: { Ans: , MgmtId: > > 176206389407942, via: 40(srv-oz-r08-n08.cenagis.local), Ver: v1, Flags: > > 10, { CreateObjectAnswer } } > > > > 2020-11-09 12:56:48,226 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (StatsCollector-5:ctx-e6db6758) (logid:451965d6) Seq > > 22-6063533948300963066: Forwarding null to 176206389407982 > > > > 2020-11-09 12:56:48,248 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) copyAsync inspecting src type SNAPSHOT copyAsync > > inspecting dest type SNAPSHOT > > > > 2020-11-09 12:56:48,249 DEBUG [c.c.a.ApiServlet] > > (qtp1497845528-1209:ctx-4e81cf12) (logid:2f3c90fe) ===START=== > > 10.89.11.31 -- GET > > command=listHosts&response=json&type=routing&listAll=true&page=1&pages > > ize=20&_=1604922838198 > > > > 2020-11-09 12:56:48,253 DEBUG [c.c.a.ApiServer] > > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) CIDRs > > from which account 'Acct[c3dd8977-12ba-11eb-9854-a0423f44350c-admin]' > > is allowed to perform API calls: 0.0.0.0/0,::/0 <http://0.0.0.0/0,::/0> > > > > 2020-11-09 12:56:48,254 WARN [c.c.a.d.ParamGenericValidationWorker] > > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) > > Received unknown parameters for command listHosts. Unknown parameters > > : listall > > > > 2020-11-09 12:56:48,254 DEBUG [c.c.a.q.QueryManagerImpl] > > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) > > >>>Searching for hosts>>> > > > > 2020-11-09 12:56:48,259 DEBUG [c.c.a.q.QueryManagerImpl] > > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) > > >>>Generating Response>>> > > > > 2020-11-09 12:56:48,293 DEBUG [c.c.a.ApiServlet] > > (qtp1497845528-1209:ctx-4e81cf12 ctx-0b68411a) (logid:2f3c90fe) > > ===END=== > > 10.89.11.31 -- GET > > command=listHosts&response=json&type=routing&listAll=true&page=1&pages > > ize=20&_=1604922838198 > > > > 2020-11-09 12:56:48,298 ERROR [o.a.c.s.e.DefaultEndPointSelector] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) IR24 select BACKUPSNAPSHOT from primary to secondary > > 13 > > dest=13 > > > > 2020-11-09 12:56:48,302 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) getCommandHostDelegation: class > > org.apache.cloudstack.storage.command.CopyCommand > > > > 2020-11-09 12:56:48,303 DEBUG [c.c.a.m.ClusteredAgentAttache] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Seq 49-6537256334104996965: Forwarding Seq > > 49-6537256334104996965: { Cmd , MgmtId: 176206389407942, via: > > 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags: 100011, > > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.a > > pache.cloudstack.storage.to <http://pache.cloudstack.storage.to> > .SnapshotObjectTO":{"path":"rbd/9945a23b-a3 > > 76-4f85-b378-9f7da8017dac/f957d3a8-702a-45aa-9e42-5222c223ef19","volum > > e":{"uuid":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeType":"ROOT", > > "dataStore":{"org.apache.cloudstack.storage.to < > http://cloudstack.storage.to> .PrimaryDataStoreTO":{"u > > uid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"poolType":"RBD","h > > ost":"rbd.cenagis.local","path":"rbd","port":6789,"url":"RBD://rbd.cen > > agis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3613-961d-f909245 > > f5477","isManaged":false}},"name":"ROOT-183","size":107374182400,"path > > ":"9945a23b-a376-4f85-b378-9f7da8017dac","volumeId":544,"vmName":"i-2- > > 183-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":54 > > 4,"deviceId":0,"cacheMode":"WRITEBACK","hypervisorType":"KVM","directD > > ownload":false},"dataStore":{"org.apache.cloudstack.storage.to.Primary > > DataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5477","id":4,"po > > olType":"RBD","host":"rbd.cenagis.local","path":"rbd","port":6789,"url > > ":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID=22d1051a-86dc-3 > > 613-961d-f909245f5477","isManaged":false}},"vmName":"i-2-183-VM","name > > ":"test","hypervisorType":"KVM","id":13,"quiescevm":false,"physicalSiz > > e":0}},"destTO":{"org.apache.cloudstack.storage.to < > http://cloudstack.storage.to> .SnapshotObjectTO":{ > > "path":"snapshots/2/544","volume":{"uuid":"9945a23b-a376-4f85-b378-9f7 > > da8017dac","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.sto > > rage.to.PrimaryDataStoreTO":{"uuid":"22d1051a-86dc-3613-961d-f909245f5 > > 477","id":4,"poolType":"RBD","host":"rbd.cenagis.local","path":"rbd"," > > port":6789,"url":"RBD://rbd.cenagis.local/rbd/?ROLE=Primary&STOREUUID= > > 22d1051a-86dc-3613-961d-f909245f5477","isManaged":false}},"name":"ROOT > > -183","size":107374182400,"path":"9945a23b-a376-4f85-b378-9f7da8017dac > > ","volumeId":544,"vmName":"i-2-183-VM","accountId":2,"format":"RAW","p > > rovisioningType":"THIN","id":544,"deviceId":0,"cacheMode":"WRITEBACK", > > "hypervisorType":"KVM","directDownload":false},"dataStore":{"com.cloud > > .agent.api.to.NfsTO":{"_url":"nfs://sys2.cenagis.local/","_role":"Imag > > e"}},"vmName":"i-2-183-VM","name":"test","hypervisorType":"KVM","id":1 > > 3,"quiescevm":false,"physicalSize":0}},"executeInSequence":false,"opti > > ons":{"fullSnapshot":"true"},"options2":{},"wait":216000}}] > > } to 176206389407982 > > > > 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request] > > (AgentManager-Handler-7:null) (logid:) Seq 49-6537256334104996965: > > Processing: { Ans: , MgmtId: 176206389407942, via: 49, Ver: v1, Flags: > > 10, > > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: > > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path","wait":0}}] } > > > > 2020-11-09 12:56:48,364 DEBUG [c.c.a.t.Request] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Seq 49-6537256334104996965: Received: { Ans: , MgmtId: > > 176206389407942, via: 49(srv-oz-r11-n11.cenagis.local), Ver: v1, Flags: > > 10, { CopyCmdAnswer } } > > > > 2020-11-09 12:56:48,373 DEBUG [c.c.s.s.SnapshotManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Failed to create > > snapshotcom.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > 2020-11-09 12:56:48,373 DEBUG [c.c.r.ResourceLimitManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Updating resource Type = snapshot count for Account = > > 2 Operation = decreasing Amount = 1 > > > > 2020-11-09 12:56:48,375 DEBUG [c.c.r.ResourceLimitManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Updating resource Type = secondary_storage count for > > Account = 2 Operation = decreasing Amount = 107374182400 > > > > 2020-11-09 12:56:48,377 ERROR [o.a.c.s.v.VolumeServiceImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Take snapshot: 544 failed > > > > com.cloud.utils.exception.CloudRuntimeException: > > com.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > at > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps > > hot(SnapshotServiceImpl.java:301) > > > > at > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS > > napshot(DefaultSnapshotStrategy.java:171) > > > > at > > com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda > > ry(SnapshotManagerImpl.java:1213) > > > > at > > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa > > nagerImpl.java:1164) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio > > n(AopUtils.java:344) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi > > npoint(ReflectiveMethodInvocation.java:198) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:163) > > > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke > > (ExposeInvocationInterceptor.java:95) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:186) > > > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic > > AopProxy.java:212) > > > > at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source) > > > > at > > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo > > lumeServiceImpl.java:2073) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V > > olumeApiServiceImpl.java:2541) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V > > olumeApiServiceImpl.java:3465) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro > > xy.java:107) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic > > eImpl.java:3471) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio > > n(AopUtils.java:344) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi > > npoint(ReflectiveMethodInvocation.java:198) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:163) > > > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke > > (ExposeInvocationInterceptor.java:95) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:186) > > > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic > > AopProxy.java:212) > > > > at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source) > > > > at > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > > > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC > > ontext(AsyncJobManagerImpl.java:603) > > > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man > > agedContextRunnable.java:49) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal > > l(DefaultManagedContext.java:56) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW > > ithContext(DefaultManagedContext.java:103) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi > > thContext(DefaultManagedContext.java:53) > > > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag > > edContextRunnable.java:46) > > > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As > > yncJobManagerImpl.java:551) > > > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor > > s.java:515) > > > > at > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > Executor.java:1128) > > > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > lExecutor.java:628) > > > > at java.base/java.lang.Thread.run(Thread.java:834) > > > > 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobHandlerProxy] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Invocation exception, caused by: > > com.cloud.utils.exception.CloudRuntimeException: > > com.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > 2020-11-09 12:56:48,378 INFO [c.c.v.VmWorkJobHandlerProxy] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897 ctx-5fb6d0f8) > > (logid:db479fbc) Rethrow exception > > com.cloud.utils.exception.CloudRuntimeException: > > com.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > 2020-11-09 12:56:48,378 DEBUG [c.c.v.VmWorkJobDispatcher] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot > > for VM 183, job origin: 1894 > > > > 2020-11-09 12:56:48,378 ERROR [c.c.v.VmWorkJobDispatcher] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Unable to complete AsyncJobVO {id:1897, userId: 2, accountId: 2, > > instanceType: null, instanceId: null, cmd: > > com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: > > rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c > > 8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20v > > Y2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMam > > F2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNj > > b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAA > > R2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAA > > AAAAAAIAAAAAAAAAt3QAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy > > 5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsC > > AAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAA1zcQB-AAcAAAAAAAACIA, > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > result: null, initMsid: 176206389407942, completeMsid: null, > lastUpdated: > > null, lastPolled: null, created: Mon Nov 09 12:56:46 CET 2020, removed: > > null}, job origin:1894 > > > > com.cloud.utils.exception.CloudRuntimeException: > > com.cloud.utils.exception.CloudRuntimeException: > > org.libvirt.LibvirtException: XML error: missing storage pool source > > path > > > > at > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnaps > > hot(SnapshotServiceImpl.java:301) > > > > at > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupS > > napshot(DefaultSnapshotStrategy.java:171) > > > > at > > com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSeconda > > ry(SnapshotManagerImpl.java:1213) > > > > at > > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotMa > > nagerImpl.java:1164) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio > > n(AopUtils.java:344) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi > > npoint(ReflectiveMethodInvocation.java:198) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:163) > > > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke > > (ExposeInvocationInterceptor.java:95) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:186) > > > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic > > AopProxy.java:212) > > > > at com.sun.proxy.$Proxy205.takeSnapshot(Unknown Source) > > > > at > > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Vo > > lumeServiceImpl.java:2073) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V > > olumeApiServiceImpl.java:2541) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(V > > olumeApiServiceImpl.java:3465) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro > > xy.java:107) > > > > at > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServic > > eImpl.java:3471) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM > > ethodAccessorImpl.java:62) > > > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del > > egatingMethodAccessorImpl.java:43) > > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio > > n(AopUtils.java:344) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi > > npoint(ReflectiveMethodInvocation.java:198) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:163) > > > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke > > (ExposeInvocationInterceptor.java:95) > > > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R > > eflectiveMethodInvocation.java:186) > > > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic > > AopProxy.java:212) > > > > at com.sun.proxy.$Proxy212.handleVmWorkJob(Unknown Source) > > > > at > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > > > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC > > ontext(AsyncJobManagerImpl.java:603) > > > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man > > agedContextRunnable.java:49) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal > > l(DefaultManagedContext.java:56) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW > > ithContext(DefaultManagedContext.java:103) > > > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi > > thContext(DefaultManagedContext.java:53) > > > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag > > edContextRunnable.java:46) > > > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As > > yncJobManagerImpl.java:551) > > > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor > > s.java:515) > > > > at > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > Executor.java:1128) > > > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > lExecutor.java:628) > > > > at java.base/java.lang.Thread.run(Thread.java:834) > > > > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Complete async job-1897, jobStatus: FAILED, resultCode: 0, result: > > rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdG > > lvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNl > > cHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cg > > ATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5n > > L1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3 > > RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNz > > ZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AHpjb20uY2xvdWQudX > > RpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbjogb3JnLmxpYnZpcnQuTGli > > dmlydEV4Y2VwdGlvbjogWE1MIGVycm9yOiBtaXNzaW5nIHN0b3JhZ2UgcG9vbCBzb3VyY2 > > UgcGF0aHVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhw > > AAAAMHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm > > 1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0Ns > > YXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTm > > FtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAABLXQAA2FwcHQAOm9yZy5hcGFj > > aGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LlNuYXBzaG90U2VydmljZUltcGx0AB > > hTbmFwc2hvdFNlcnZpY2VJbXBsLmphdmF0AA5iYWNrdXBTbmFwc2hvdHBwc3EAfgAMAQAA > > AKtxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LkRlZm > > F1bHRTbmFwc2hvdFN0cmF0ZWd5dAAcRGVmYXVsdFNuYXBzaG90U3RyYXRlZ3kuamF2YXEA > > fgARcHBzcQB-AAwBAAAEvXEAfgAOdAAuY29tLmNsb3VkLnN0b3JhZ2Uuc25hcHNob3QuU2 > > 5hcHNob3RNYW5hZ2VySW1wbHQAGFNuYXBzaG90TWFuYWdlckltcGwuamF2YXQAGWJhY2t1 > > cFNuYXBzaG90VG9TZWNvbmRhcnlwcHNxAH4ADAEAAASMcQB-AA5xAH4AFnEAfgAXdAAMdG > > FrZVNuYXBzaG90cHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRp > > dmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YX > > QAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAYxMS4wLjhzcQB-AAwCAAAAPnBxAH4AHHEAfgAd > > dAAGaW52b2tlcQB-AB9xAH4AIHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZW > > N0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNj > > ZXNzb3JJbXBsLmphdmFxAH4AInEAfgAfcQB-ACBzcQB-AAwCAAACNnB0ABhqYXZhLmxhbm > > cucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAicQB-AB9xAH4AIHNxAH4ADAEA > > AAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdA > > ANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EA > > fgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZW > > ZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9u > > LmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4ALnEAfgAvdA > > AHcHJvY2VlZHBwc3EAfgAMAQAAAF9xAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9w > > LmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm > > 9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-ACJwcHNxAH4ADAEAAAC6cQB-AA5xAH4ALnEA > > fgAvcQB-ADJwcHNxAH4ADAEAAADUcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC > > 5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5Lmph > > dmFxAH4AInBwc3EAfgAMAP____9wdAAXY29tLnN1bi5wcm94eS4kUHJveHkyMDVwcQB-AB > > pwcHNxAH4ADAEAAAgZcQB-AA50ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52 > > b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-AB > > pwcHNxAH4ADAEAAAntcQB-AA50ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2 > > aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZVRha2 > > VWb2x1bWVTbmFwc2hvdHBwc3EAfgAMAQAADYlxAH4ADnEAfgBAcQB-AEFxAH4AQnBwc3EA > > fgAMAv____5wcQB-ABxxAH4AHXEAfgAecQB-AB9xAH4AIHNxAH4ADAIAAAA-cHEAfgAccQ > > B-AB1xAH4AInEAfgAfcQB-ACBzcQB-AAwCAAAAK3BxAH4AJHEAfgAlcQB-ACJxAH4AH3EA > > fgAgc3EAfgAMAgAAAjZwcQB-ACdxAH4AKHEAfgAicQB-AB9xAH4AIHNxAH4ADAEAAABrcQ > > B-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9i > > SGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAAA2PcQB-AA > > 5xAH4AQHEAfgBBcQB-AEtwcHNxAH4ADAL____-cHEAfgAccQB-AB1xAH4AHnEAfgAfcQB- > > ACBzcQB-AAwCAAAAPnBxAH4AHHEAfgAdcQB-ACJxAH4AH3EAfgAgc3EAfgAMAgAAACtwcQ > > B-ACRxAH4AJXEAfgAicQB-AB9xAH4AIHNxAH4ADAIAAAI2cHEAfgAncQB-AChxAH4AInEA > > fgAfcQB-ACBzcQB-AAwBAAABWHEAfgAOcQB-ACpxAH4AK3EAfgAscHBzcQB-AAwBAAAAxn > > EAfgAOcQB-AC5xAH4AL3EAfgAwcHBzcQB-AAwBAAAAo3EAfgAOcQB-AC5xAH4AL3EAfgAy > > cHBzcQB-AAwBAAAAX3EAfgAOcQB-ADRxAH4ANXEAfgAicHBzcQB-AAwBAAAAunEAfgAOcQ > > B-AC5xAH4AL3EAfgAycHBzcQB-AAwBAAAA1HEAfgAOcQB-ADhxAH4AOXEAfgAicHBzcQB- > > AAwA_____3B0ABdjb20uc3VuLnByb3h5LiRQcm94eTIxMnBxAH4AS3Bwc3EAfgAMAQAAAG > > ZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9i > > RGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACW3EAfgAOdAA_b3JnLmFwYW > > NoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1w > > bCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AA > > wBAAAAMXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5N > > YW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdm > > F0AANydW5wcHNxAH4ADAEAAAA4cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu > > YWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TW > > FuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABncQB-AA50AEBvcmcuYXBh > > Y2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb2 > > 50ZXh0cQB-AGd0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA1cQB-AA5xAH4AanEA > > fgBndAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAucQB-AA50ADxvcmcuYXBhY2hlLm > > Nsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4A > > Y3EAfgBkcHBzcQB-AAwBAAACJ3EAfgAOcQB-AF5xAH4AX3EAfgBkcHBzcQB-AAwCAAACA3 > > B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAO > > RXhlY3V0b3JzLmphdmFxAH4AaHEAfgAfcQB-ACBzcQB-AAwCAAABCHB0AB9qYXZhLnV0aW > > wuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AGRxAH4AH3EA > > fgAgc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZW > > N1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AH3EAfgAg > > c3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG > > 9yJFdvcmtlcnEAfgB5cQB-AGRxAH4AH3EAfgAgc3EAfgAMAgAAA0JwdAAQamF2YS5sYW5n > > LlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AGRxAH4AH3EAfgAgc3IAH2phdmEudXRpbC5Db2 > > xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA > > > > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Publish async job-1897 complete on message bus > > > > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Wake up jobs related to job-1897 > > > > 2020-11-09 12:56:48,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Update db status for job-1897 > > > > 2020-11-09 12:56:48,381 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Wake up jobs joined with job-1897 and disjoin all subjobs created from > > job- 1897 > > > > 2020-11-09 12:56:48,385 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-1897 > > > > 2020-11-09 12:56:48,385 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (Work-Job-Executor-26:ctx-15ef9a01 job-1894/job-1897) (logid:db479fbc) > > Remove job-1897 from job monitoring > > > > 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Complete > > async job-1894, jobStatus: FAILED, resultCode: 530, result: > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > > [],"errorcode":530,"errortext":"Failed > > to create snapshot due to an internal error creating snapshot for > > volume 9945a23b-a376-4f85-b378-9f7da8017dac"} > > > > 2020-11-09 12:56:48,392 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (API-Job-Executor-35:ctx-326b32af job-1894) (logid:db479fbc) Publish > > async > > job-1894 complete on message bus > > > > > > > > > -- > > > > Andrija Panić > >
