Hi Andrei, Have you checked the 'status' and 'removed' timestamp in snapshots table, and 'state' in snapshot_store_ref table for these snapshots.
Similar issue logged (by Ed, as mentioned in his email) here: https://github.com/apache/cloudstack/issues/4797. Is it the same issue? Regards, Suresh On 17/06/21, 2:18 PM, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: Hi Suresh, Please see below the answers to your questions. ----- Original Message ----- > From: "Suresh Anaparti" <suresh.anapa...@shapeblue.com> > To: "users" <users@cloudstack.apache.org> > Sent: Thursday, 17 June, 2021 06:36:27 > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > Hi Andrei, > > Can you check if the storage garbage collector is enabled or not in your env > (specified using the global setting 'storage.cleanup.enabled'). If it is > enabled, check the interval & delay setting: 'storage.cleanup.interval' and > 'storage.cleanup.delay', and see the logs to confirm cleanup is performed or > not. storage.cleanup.enabled is true storage.cleanup.interval is 3600 storage.cleanup.delay is 360086400 > > Also, check the snapshot status / state in snapshots & snapshot_store_ref tables > for the snapshots that are not deleted during the cleanup. Is 'removed' > timestamp set for them in snapshots table? > The table snapshots has NULL under the removed column in all snapshots that I've removed. The table snapshot_store_ref has no such column, but the state shown as Destroyed. > Regards, > Suresh > >On 16/06/21, 9:46 PM, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > Hello, > > I've done some more investigation and indeed, the snapshots were not taken > because the secondary storage was over 90% used. I have started cleaning some > of the older volumes and noticed another problem. After removing snapshots, > they do not seem to be removed from the secondary storage. I've removed all > snapshots over 24 hours ago and it looks like the disk space hasn't been freed > up at all. > > Looks like there are issues with snapshotting function after all. > > Andrei > > > > > > > ----- Original Message ----- > > From: "Harikrishna Patnala" <harikrishna.patn...@shapeblue.com> > > To: "users" <users@cloudstack.apache.org> > > Sent: Tuesday, 8 June, 2021 03:33:57 > > Subject: Re: Snapshots are not working after upgrading to 4.15.0 > > > Hi Andrei, > > > > Can you check the following things and let us know? > > > > > > 1. Can you try creating a new volume and then create snapshot of that, to check > > if this an issue with old entries > > 2. For the snapshots which are failing can you check if you are seeing any > > error messages like this "Can't find an image storage in zone with less than". > > This is to check if secondary storage free space check failed. > > 3. For the snapshots which are failing and if it is delta snapshot can you > > check if its parent's snapshot entry exists in "snapshot_store_ref" table with > > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". This is > > to find the secondary storage where the parent snapshot backup is located. > > > > Regards, > > Harikrishna > > ________________________________ > > From: Andrei Mikhailovsky <and...@arhont.com.INVALID> > > Sent: Monday, June 7, 2021 7:00 PM > > To: users <users@cloudstack.apache.org> > > Subject: Snapshots are not working after upgrading to 4.15.0 > > > > Hello everyone, > > > > I am having an issue with volume snapshots since I've upgraded to 4.15.0. None > > of the volumes are being snapshotted regardless if the snapshot is initiated > > manually or from the schedule. The strange thing is that if I manually take the > > snapshot, the GUI shows Success status, but the Storage>Snapshots show an Error > > status. Here is what I see in the management server logs: > > > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Done > > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143 > > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Remove > > job-86143 from job monitoring > > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy snapshot > > com.cloud.utils.exception.CloudRuntimeException: can not find an image stores > > at > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271) > > at > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171) > > at > > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > 2021-06-07 13:55:20,152 DEBUG [c.c.s.s.SnapshotManagerImpl] > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Backing up of snapshot > > failed, for snapshot with ID 53531, left with 2 more attempts > > > > > > I've checked and the Secondary storage is configured and visible in the GUI. I > > can also mount it manually from the management server and a couple of host > > servers that I've tested. In addition, I can successfully upload an ISO image > > and that registers just fine and I can create new VMs using the newly uploaded > > ISO image. > > > > I've had no such problems with 4.13.x ACS, so the issue seems to have been > > introduced after doing the upgrade to 4.15.0. > > > > Could you please let me know how do I fix the issue? > > > > Cheers > > > > andrei
