Extending these timeouts in the “wait” configs seems to have helped. One of my 1TB volumes is finally migrating.
WHat’s I’ve noticed is if I allocate a new 1TB volume, I can migrate this between NFS and Ceph and it takes only about a 1 minute. I assume this is because it’s “thin provisioned” and there’s no actual data on the volume. But these other volumes I’m trying to move are also “thin provisioned” but they’re a part of a LVM group. Does making a thin provisioned device part of a LVM group defeat the thin provisioning? I know these volumes weren’t full, but I thought perhaps there’s a chance that since it’s a pv in a LVM config, that maybe that defeats the thin provisioning and it counts it as a full 1TB volume? I’m just spitballing but I’m trying to understand how this works so we can do the right thing when provisioning additional volumes. Also, the behavior I’m seeing is it takes a very long time before I see the block image show up on the Ceph side. Perhaps it preallocated a image before copying the data? But it seemed strange that I wouldn’t immidiately see the image appear on the Ceph side after initiating a migration. It’s hard to see what’s actually going on from the logs and the interface. Thanks -jeremy > On Tuesday, Jan 16, 2024 at 11:29 PM, Jeremy Hansen <jer...@skidrow.la > (mailto:jer...@skidrow.la)> wrote: > I changed copy.volume.wait to 72000 > > But I just noticed: > > kvm.storage.online.migration.wait and kvm.storage.offline.migration.wait. > Worth changing this? > > Thanks > -jeremy > > > > On Tuesday, Jan 16, 2024 at 11:01 PM, Jithin Raju > > <jithin.r...@shapeblue.com (mailto:jithin.r...@shapeblue.com)> wrote: > > Hi Jeremy, > > > > Have you checked the ‘wait’ parameter? Used as wait * 2 timeout. > > > > -Jithin > > > > From: Jeremy Hansen <jer...@skidrow.la.INVALID> > > Date: Wednesday, 17 January 2024 at 12:14 PM > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > Subject: Re: Issues migrating primary storage > > Unfortunately the upgrade didn’t help: > > > > Resource [StoragePool:3] is unreachable: Volume > > [{"name”:”bigdisk","uuid":"8f24b8a6-229a-4311-9ddc-d6c6acb89aca"}] > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException: > > Failed to copy > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/8f24b8a6-229a-4311-9ddc-d6c6acb89aca > > to 5837f4e6-9307-43a9-a50c-8c9c885f25e8.qcow2]. > > > > > > > > Anything else I can try? I’m trying to move away from NFS completely. > > > > -jeremy > > > > > > > > > > > > > > On Tuesday, Jan 16, 2024 at 7:06 AM, Suresh Kumar Anaparti > > <sureshkumar.anapa...@gmail.com<mailto:sureshkumar.anapa...@gmail.com>> > > wrote: > > Hi Jeremy, > > > > Can you extend with the config 'migratewait' and check. > > > > Regards, > > Suresh > > > > On Tue, Jan 16, 2024 at 1:45 PM Jeremy Hansen <jer...@skidrow.la.invalid> > > wrote: > > > > > > I have some large volumes I’m trying to migrate from NFS to Ceph/RBD. 1TB > > volumes. These inevitably times out. I extended these configs: > > > > copy.volume.wait=72000 > > job.cancel.threshold.minutes=480 > > job.expire.minutes=1440 > > > > This helped with smaller volumes but large once still eventually fail. > > > > 2024-01-16 07:50:25,929 DEBUG [c.c.a.t.Request] > > (AgentManager-Handler-8:null) (logid:) Seq 1-5583619113009291196: > > Processing: { Ans: , MgmtId: 20558852646968, via: 1, Ver: v1, Flags: 10, > > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: > > Failed to copy > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e > > to > > b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2","wait":"0","bypassHostMaintenance":"false"}}] > > } > > > > 2024-01-16 07:50:26,698 DEBUG [c.c.s.VolumeApiServiceImpl] > > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b) > > (logid:d7d98b81) Failed to migrate volume > > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] > > is unreachable: Volume > > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}] > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException: > > Failed to copy > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e > > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2]. > > at > > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1348) > > at jdk.internal.reflect.GeneratedMethodAccessor672.invoke(Unknown > > Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > at com.sun.proxy.$Proxy227.migrateVolume(Unknown Source) > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3356) > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721) > > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown > > Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735) > > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown > > Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source) > > at > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2024-01-16 07:50:26,727 ERROR [c.c.v.VmWorkJobHandlerProxy] > > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176 ctx-bc7b188b) > > (logid:d7d98b81) Invocation exception, caused by: > > com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:3] > > is unreachable: Volume > > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}] > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException: > > Failed to copy > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e > > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2]. > > > > com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:3] > > is unreachable: Volume > > [{"name":"sequencingdata","uuid":"861a6692-e746-4401-9cda-bd791b7d3b5e"}] > > migration failed due to [com.cloud.utils.exception.CloudRuntimeException: > > Failed to copy > > /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/861a6692-e746-4401-9cda-bd791b7d3b5e > > to b7acadc8-34a1-4d7a-8040-26368dafc21d.qcow2]. > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3363) > > at > > com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:4721) > > at jdk.internal.reflect.GeneratedMethodAccessor671.invoke(Unknown > > Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:4735) > > at jdk.internal.reflect.GeneratedMethodAccessor670.invoke(Unknown > > Source) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > at com.sun.proxy.$Proxy232.handleVmWorkJob(Unknown Source) > > at > > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at > > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2024-01-16 07:50:26,744 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-41:ctx-e5baf6dc job-1175/job-1176) (logid:d7d98b81) > > Complete async job-1176, jobStatus: FAILED > > > > This is 4.18.0.0. > > > > Is there other timeout that might be at play here? StoragePool:3] is > > unreachable. Is StoragePool:3 referring to the NFS server or RBD? Or how > > do I interpret StoragePool:3 and why it thinks it’s unavailable? > > > > Thanks > > -jeremy > > > > > >
signature.asc
Description: PGP signature
