Hi, Andrei, After a quick check of the code, I guess that the backup of the snapshot fails because the secondary storage space should be less than 90%, and you pointed that yours is 90.6% full.
Regards, Slavka On Mon, Jun 14, 2021 at 1:10 PM Andrei Mikhailovsky <and...@arhont.com.invalid> wrote: > Hi Harikrishna, > > I've done some more testing testing just now. Please see below the answers > to your questions/comments: > > > > ----- Original Message ----- > > From: "Harikrishna Patnala" <harikrishna.patn...@shapeblue.com> > > To: "users" <users@cloudstack.apache.org> > > Sent: Tuesday, 8 June, 2021 03:33:57 > > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > > > Hi Andrei, > > > > Can you check the following things and let us know? > > > > > > 1. Can you try creating a new volume and then create snapshot of that, > to check > > if this an issue with old entries > > AM: I've done testing with a newly created image which was attached to a > vm. The same problem happens with new images as well as old images. Please > see the link below for the management server log. > > > https://zerobin.net/?5781e4b65d9e3605#+GtIC7JBtp70Q0cw65cypJDiSyba/r/JldRsAyOI8l4= > > > > 2. For the snapshots which are failing can you check if you are seeing > any > > error messages like this "Can't find an image storage in zone with less > than". > > This is to check if secondary storage free space check failed. > > > AM: I do not see any such message in the logs. tried grepping the logs for > a couple of weeks and nothing comes up. Having said this, the secondary > storage is about 90.6% full. > > > > 3. For the snapshots which are failing and if it is delta snapshot can > you > > check if its parent's snapshot entry exists in "snapshot_store_ref" > table with > > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". > This is > > to find the secondary storage where the parent snapshot backup is > located. > > > > AM: all snapshots are failing, not just selective few. Some volumes are > brand new, as I've indicated above, others do have previous snapshots. I > only have a single secondary storage, so all snaps should be in one place. > > > > > > Regards, > > Harikrishna > > ________________________________ > > From: Andrei Mikhailovsky <and...@arhont.com.INVALID> > > Sent: Monday, June 7, 2021 7:00 PM > > To: users <users@cloudstack.apache.org> > > Subject: Snapshots are not working after upgrading to 4.15.0 > > > > Hello everyone, > > > > I am having an issue with volume snapshots since I've upgraded to > 4.15.0. None > > of the volumes are being snapshotted regardless if the snapshot is > initiated > > manually or from the schedule. The strange thing is that if I manually > take the > > snapshot, the GUI shows Success status, but the Storage>Snapshots show > an Error > > status. Here is what I see in the management server logs: > > > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) > Done > > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 > > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) > Remove > > job-86143 from job monitoring > > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy > snapshot > > com.cloud.utils.exception.CloudRuntimeException: can not find an image > stores > > at > > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) > > at > > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > > at > > > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > > at > > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of > snapshot > > failed, for snapshot with ID 53531, left with 2 more attempts > > > > > > I've checked and the Secondary storage is configured and visible in the > GUI. I > > can also mount it manually from the management server and a couple of > host > > servers that I've tested. In addition, I can successfully upload an ISO > image > > and that registers just fine and I can create new VMs using the newly > uploaded > > ISO image. > > > > I've had no such problems with 4.13.x ACS, so the issue seems to have > been > > introduced after doing the upgrade to 4.15.0. > > > > Could you please let me know how do I fix the issue? > > > > Cheers > > > > andrei >
