Failure creating a template from a snapshot
I see this error: Create template (GSA Security Scanner) Failed to copy snapshot:java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: copySnapshotAsyncCallback Any clues on this? Thanks -jeremy signature.asc Description: Message signed with OpenPGP
Re: Error trying to create a volume snapshot
Also, I noticed when I initiate the snapshot from the Cloudstack side, an image does get allocated from Ceph’s perspective but then it ultimately fails but the snapshot remains on the ceph side. -jeremy > On Jun 15, 2021, at 5:52 PM, Jeremy Hansen wrote: > > Thank you. Creating the snapshot on the Ceph side seems to be no problem and > pretty much instantaneous. It’s just odd since I’m able to snapshot other VM > root volumes also residing on the ceph rbd pool. > > -jeremy > > >> On Jun 15, 2021, at 3:24 AM, Andrija Panic wrote: >> >> I would focus on the actual qemu-img error (i.e. nothing to do with >> CloudStack, but something to do with Ceph+RBD - try to do internal Ceph >> snapshot, see if the image is corrupted or not, etc, etc): >> >> QemuImgException: qemu-img: error while writing sector 7921663: >> Input/output errorqemu-img: Failed to flush the L2 table cache: >> Input/output errorqemu-img: Failed to flush the refcount block cache: >> Input/output error >> >> Perhaps someone else have seen something similar. >> >> >> On Tue, 15 Jun 2021 at 09:32, Jeremy Hansen wrote: >> >>> >>> For some reason a particular snapshot I’m trying to create for a volume >>> fails and I see this error: >>> >>> 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] >>> (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) >>> (logid:da060faf) Take snapshot: 49 failed >>> com.cloud.utils.exception.CloudRuntimeException: >>> org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while >>> writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 >>> table cache: Input/output errorqemu-img: Failed to flush the refcount block >>> cache: Input/output error >>> at >>> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) >>> at >>> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) >>> at >>> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) >>> at >>> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>> at >>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>> at >>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) >>> at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) >>> at >>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>> at >>> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >>> Method) >>> at >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>> at >>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>> at >>>
Re: Error trying to create a volume snapshot
Thank you. Creating the snapshot on the Ceph side seems to be no problem and pretty much instantaneous. It’s just odd since I’m able to snapshot other VM root volumes also residing on the ceph rbd pool. -jeremy > On Jun 15, 2021, at 3:24 AM, Andrija Panic wrote: > > I would focus on the actual qemu-img error (i.e. nothing to do with > CloudStack, but something to do with Ceph+RBD - try to do internal Ceph > snapshot, see if the image is corrupted or not, etc, etc): > > QemuImgException: qemu-img: error while writing sector 7921663: > Input/output errorqemu-img: Failed to flush the L2 table cache: > Input/output errorqemu-img: Failed to flush the refcount block cache: > Input/output error > > Perhaps someone else have seen something similar. > > > On Tue, 15 Jun 2021 at 09:32, Jeremy Hansen wrote: > >> >> For some reason a particular snapshot I’m trying to create for a volume >> fails and I see this error: >> >> 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] >> (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) >> (logid:da060faf) Take snapshot: 49 failed >> com.cloud.utils.exception.CloudRuntimeException: >> org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while >> writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 >> table cache: Input/output errorqemu-img: Failed to flush the refcount block >> cache: Input/output error >>at >> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) >>at >> org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) >>at >> com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) >>at >> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) >>at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) >>at >> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) >>at >> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) >>at >> com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>at >> com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >>at >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>at >> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.base/java.lang.reflect.Method.invoke(Method.java:566) >>at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) >>at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) >>at >>
Unable to add template to new deployment
I've setup ACS 4.15 and created the first zone on it using the wizard through primate. The zone is enabled and isn't showing any issues in the UI. I can see two system vm's running (secondarystoragevm and consoleproxy) and can SSH into the VM's from the compute node. However when I try to add a template through the UI I get the following error: There is no secondary storage VM for downloading template to image store LXC_SEC_STOR1 And on the controller I can see the corresponding log entries when I try to submit the new template: 2021-06-15 22:13:35,884 DEBUG [c.c.a.ApiServlet] (qtp1644231115-348:ctx-aab81632) (logid:ac328268) ===START=== 172.16.44.18 -- GET name=Ubuntu+20.04=Ubuntu+20.04+(Focal)+64-bit=9f6f5b49-0e12-4af4-a13b-2ace6c47de43=LXC=TAR=aa18f3ad-cd73-11eb-b1da-5254008f72d5=true=getUploadParamsForTemplate=json 2021-06-15 22:13:35,924 DEBUG [c.c.a.ApiServer] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) CIDRs from which account 'Acct[f8d6949d-cd74-11eb-b1da-5254008f72d5-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2021-06-15 22:13:36,093 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) template 203 is not in store:1, type:Image 2021-06-15 22:13:36,138 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) template 203 is already in store:1, type:Image 2021-06-15 22:13:36,142 WARN [c.c.t.HypervisorTemplateAdapter] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) There is no secondary storage VM for downloading template to image store LXC_SEC_STOR1 2021-06-15 22:13:36,146 DEBUG [c.c.u.d.T.Transaction] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) Rolling back the transaction: Time = 98 Name = qtp1644231115-348; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-Transaction.execute:47-HypervisorTemplateAdapter.createTemplateForPostUpload:298-TemplateManagerImpl.registerPostUploadInternal:361-TemplateManagerImpl.registerTemplateForPostUpload:423-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566 2021-06-15 22:13:36,229 ERROR [c.c.a.ApiServer] (qtp1644231115-348:ctx-aab81632 ctx-da7281d3) (logid:ac328268) unhandled exception executing api command: [Ljava.lang.String;@2ad13bf5 com.cloud.utils.exception.CloudRuntimeException: There is no secondary storage VM for downloading template to image store LXC_SEC_STOR1 at com.cloud.template.HypervisorTemplateAdapter$1.doInTransaction(HypervisorTemplateAdapter.java:363) at com.cloud.template.HypervisorTemplateAdapter$1.doInTransaction(HypervisorTemplateAdapter.java:298) at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50) at com.cloud.utils.db.Transaction.execute(Transaction.java:40) at com.cloud.utils.db.Transaction.execute(Transaction.java:47) at com.cloud.template.HypervisorTemplateAdapter.createTemplateForPostUpload(HypervisorTemplateAdapter.java:298) at com.cloud.template.TemplateManagerImpl.registerPostUploadInternal(TemplateManagerImpl.java:361) at com.cloud.template.TemplateManagerImpl.registerTemplateForPostUpload(TemplateManagerImpl.java:423) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy198.registerTemplateForPostUpload(Unknown Source) at
RE: Alter Shared Guest Network?
FYI tested this on 4.15 with specifics: - Shared network with 2 ip range for example 10.10.10.10 - 10.10.10.11 - created as much VMs as ACS allows me which is 1 (first ip gets assigned to the VR) - expanded the the range of the shared network in table "VLAN" from 10.10.10.10-10.10.10.11 to 10.10.10.10-10.10.10.12 - Dublicated existing entry in table "user_ip_address" for ip in that specific shared network. Changed the following columns with new entries: --- ID to the next unreserved --- UUID to unique one for the table --- public_ip_address to 10.10.10.12 --- allocated - make it NULL --- state - make it Free --- mac_address - look at the whole table and set it to the next one that is not used Back to ACS gui I can create a new VM in that network and Ip is assigned. But there are some underwater stones that are created this way. As IDs are created manually ACS DB is not updating its sequence so I was wondering if new network is created would it take the same MAC ID. After creating a new network and looking again in the table - the answer to this question is yes - https://imgur.com/YnGMGRE. So besides the 2 tables another one should be edited but so far I cannot find where is the sequence kept. Best regards, Jordan -Original Message- From: Andrija Panic Sent: Monday, June 14, 2021 10:24 PM To: users Subject: Re: Alter Shared Guest Network? [X] This message came from outside your organization ANother is is the, if not mistaken, the VLAN table. which will contain the range as x.x.x.1-x.x.x.10 - etc - this is needed to be updated as well (if you manually add records in the user_ip_address table) best, On Thu, 10 Jun 2021 at 18:23, Jeremy Hansen wrote: > Thanks. I’ll take a look table. > > -jeremy > > > On Jun 10, 2021, at 6:57 AM, Yordan Kostov wrote: > > > > Hello Jeremy, > > > >Once a shared network with DHCP offering is created the IPs > > fitting > into the defined range are created in table called "user_ip_address". > >They are created one by one so if range between x.x.x.x.11 and > x.x.x.210 is created this will add 200 entries. So if you want to > expand that you need to add more entries manually, which is a bit unfortunate. > > > > Best regards, > > Jordan > > > > -Original Message- > > From: Jeremy Hansen > > Sent: Thursday, June 10, 2021 12:12 AM > > To: users@cloudstack.apache.org > > Subject: Re: Alter Shared Guest Network? > > > > > > [X] This message came from outside your organization > > > > > >> On Jun 9, 2021, at 1:39 PM, Wido den Hollander wrote: > >> > >> > >> > On 6/9/21 3:55 PM, Jeremy Hansen wrote: > >>> When I created my shared network config, I specified too narrow of > >>> an > IP range. > >>> > >>> I can’t seem to figure out how to alter this config via the web > interface. Is this possible? > >>> > >> > >> Not via de UI nor API. You will need to hack this in the database. > >> Or remove the network and create it again. But this is only > >> possible if there are no VMs in the network. > >> > >> Wido > > > > Thanks, recreating it seems like the easiest option since I’m only > > in > testing phase right now, but I’m curious what it would take to alter > tables to fix this. Any clues as to what tables/fields would need to be > updated? > > > >> > >>> -jeremy > >>> > > > > -- Andrija Panić
Re: Error trying to create a volume snapshot
I would focus on the actual qemu-img error (i.e. nothing to do with CloudStack, but something to do with Ceph+RBD - try to do internal Ceph snapshot, see if the image is corrupted or not, etc, etc): QemuImgException: qemu-img: error while writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 table cache: Input/output errorqemu-img: Failed to flush the refcount block cache: Input/output error Perhaps someone else have seen something similar. On Tue, 15 Jun 2021 at 09:32, Jeremy Hansen wrote: > > For some reason a particular snapshot I’m trying to create for a volume > fails and I see this error: > > 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] > (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) > (logid:da060faf) Take snapshot: 49 failed > com.cloud.utils.exception.CloudRuntimeException: > org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while > writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 > table cache: Input/output errorqemu-img: Failed to flush the refcount block > cache: Input/output error > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) > at > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) > at > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) > at > com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > at com.sun.proxy.$Proxy215.handleVmWorkJob(Unknown Source) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at >
Re: Rebooted and now I see unable to find storage pool
Thank you. Looks like a good document. -jeremy > On Jun 14, 2021, at 3:33 PM, Andrija Panic wrote: > > https://www.shapeblue.com/ceph-and-cloudstack-part-1/ (there are 3 parts...) > > You should also read on the client-side caching, etc (on KVM hosts). > > Best, > > On Fri, 11 Jun 2021 at 14:16, Jeremy Hansen wrote: > >> Turns out my other two vm hosts didn’t have rbd/ceph libs installed. This >> fixed my issue. >> >> -jeremy >> >>> On Jun 10, 2021, at 6:50 PM, Jeremy Hansen wrote: >>> >>> 2021-06-10 18:46:19,665 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create RBD storage >> pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does >> the pool 'rbd' exist?: No such file or directory >>> 2021-06-10 18:46:19,666 ERROR [kvm.storage.LibvirtStorageAdaptor] >> (agentRequest-Handler-5:null) (logid:34e88890) Failed to create the RBD >> storage pool, cleaning up the libvirt secret >>> >>> I should mention that I’ve defined a Ceph RBD primary storage volume. >> Disabling RBD allowed the vm hosts to rejoin the cluster, but I’d like to >> understand what happened here as I plan on using Ceph RBD as my primary >> storage. >>> >>> Thanks >>> -jeremy >>> >>> On Jun 10, 2021, at 6:45 PM, Jeremy Hansen wrote: I removed all of my VMs and all volumes. I rebooted all the servers >> involved in my cluster and now I see this: 2021-06-10 18:41:38,824 WARN [cloud.agent.Agent] >> (agentRequest-Handler-2:null) (logid:4f4da278) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create >> storage pool: a6768f2e-3e3c-3aad-938e-83a9efb6deab at >> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:645) at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:329) at >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:323) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35) at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) and too of my VM hosts are unable to connect. How do I resolve this situation? How did I lose a storage pool? Thanks -jeremy >>> >> >> > > -- > > Andrija Panić signature.asc Description: Message signed with OpenPGP
Error trying to create a volume snapshot
For some reason a particular snapshot I’m trying to create for a volume fails and I see this error: 2021-06-15 00:28:03,416 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-11:ctx-a1408657 job-673/job-674 ctx-5db754de) (logid:da060faf) Take snapshot: 49 failed com.cloud.utils.exception.CloudRuntimeException: org.apache.cloudstack.utils.qemu.QemuImgException: qemu-img: error while writing sector 7921663: Input/output errorqemu-img: Failed to flush the L2 table cache: Input/output errorqemu-img: Failed to flush the refcount block cache: Input/output error at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301) at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1215) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1166) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy208.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:2087) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2638) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:3627) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3633) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy215.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at
Apache CloudStack YouTube Channel
Hello all, Our community reached a small milestone today - 200 YouTube Subscribers! Can we make them 300 soon? If you still have not subscribed, please do so, so that you can get alerts for the newest videos. Some new interviews with community members are coming! Kind regards,
