[ovirt-users] Deleting large snapshots blocks the whole cluster
Hi, we have some really large snapshots left from a migration. Since our store is almost full now we have to delete them now. Some snapshots are around 1TB already. The last time we tried to delete a ~500GB snapshot the delete task blocked the whole diskstore's IO and the whole cluster and all hosts became unavailable. Is there a way to delete large snapshots in a humane way that will not block everything? Cheers, Stefan signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Deleting large snapshots blocks the whole cluster
Do you see swapping on the SPM? If yes a regular echo 3 drop_caches could help. Markus Am 27.10.2014 10:57 schrieb Stefan Wendler stefan.wend...@tngtech.com: Hi, we have some really large snapshots left from a migration. Since our store is almost full now we have to delete them now. Some snapshots are around 1TB already. The last time we tried to delete a ~500GB snapshot the delete task blocked the whole diskstore's IO and the whole cluster and all hosts became unavailable. Is there a way to delete large snapshots in a humane way that will not block everything? Cheers, Stefan Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Deleting large snapshots blocks the whole cluster
Hi, do you mean during snapshot deletion or in general? In general we do not have any swap usage at all. On none of the 4 hosts. We had some swap I/O on the host that is SPM most of the time. But not during that period where we tried to delete the snapshot. Cheers, Stefan On 10/27/14 11:08, Markus Stockhausen wrote: Do you see swapping on the SPM? If yes a regular echo 3 drop_caches could help. Markus Am 27.10.2014 10:57 schrieb Stefan Wendler stefan.wend...@tngtech.com: Hi, we have some really large snapshots left from a migration. Since our store is almost full now we have to delete them now. Some snapshots are around 1TB already. The last time we tried to delete a ~500GB snapshot the delete task blocked the whole diskstore's IO and the whole cluster and all hosts became unavailable. Is there a way to delete large snapshots in a humane way that will not block everything? Cheers, Stefan signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Deleting large snapshots blocks the whole cluster
Von: Stefan Wendler [stefan.wend...@tngtech.com] Gesendet: Montag, 27. Oktober 2014 11:39 An: Markus Stockhausen Cc: users@ovirt.org Betreff: Re: [ovirt-users] Deleting large snapshots blocks the whole cluster Hi, do you mean during snapshot deletion or in general? In general we do not have any swap usage at all. On none of the 4 hosts. We had some swap I/O on the host that is SPM most of the time. But not during that period where we tried to delete the snapshot. I thought of the bug described in RH bugzilla 1138690 [SCALE] snapshot deletion - heavy swapping on SPM. In case you have no access to it (as it is RHEV tagged): qemu-img reads through pagecache during snapshot deletion. A new flag has been intrduced in August this year that allows to avoid page cache. This flag must be provided by OVirt/RHEV. Target release is 3.6. Until then the only bugfix is to drop pagecache manually on regular intervals on the hypversior and especially the SPM node. Markus Cheers, Stefan On 10/27/14 11:08, Markus Stockhausen wrote: Do you see swapping on the SPM? If yes a regular echo 3 drop_caches could help. Markus Am 27.10.2014 10:57 schrieb Stefan Wendler stefan.wend...@tngtech.com: Hi, we have some really large snapshots left from a migration. Since our store is almost full now we have to delete them now. Some snapshots are around 1TB already. The last time we tried to delete a ~500GB snapshot the delete task blocked the whole diskstore's IO and the whole cluster and all hosts became unavailable. Is there a way to delete large snapshots in a humane way that will not block everything? Cheers, Stefan Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Deleting large snapshots blocks the whole cluster
I see. Then I will keep an eye on this tonight where we have a huge all ovirt downtime just to delete those snapshots ^^ Thanks and Cheers, Stefan On 10/27/14 13:07, Markus Stockhausen wrote: Von: Stefan Wendler [stefan.wend...@tngtech.com] Gesendet: Montag, 27. Oktober 2014 11:39 An: Markus Stockhausen Cc: users@ovirt.org Betreff: Re: [ovirt-users] Deleting large snapshots blocks the whole cluster Hi, do you mean during snapshot deletion or in general? In general we do not have any swap usage at all. On none of the 4 hosts. We had some swap I/O on the host that is SPM most of the time. But not during that period where we tried to delete the snapshot. I thought of the bug described in RH bugzilla 1138690 [SCALE] snapshot deletion - heavy swapping on SPM. In case you have no access to it (as it is RHEV tagged): qemu-img reads through pagecache during snapshot deletion. A new flag has been intrduced in August this year that allows to avoid page cache. This flag must be provided by OVirt/RHEV. Target release is 3.6. Until then the only bugfix is to drop pagecache manually on regular intervals on the hypversior and especially the SPM node. Markus Cheers, Stefan On 10/27/14 11:08, Markus Stockhausen wrote: Do you see swapping on the SPM? If yes a regular echo 3 drop_caches could help. Markus Am 27.10.2014 10:57 schrieb Stefan Wendler stefan.wend...@tngtech.com: Hi, we have some really large snapshots left from a migration. Since our store is almost full now we have to delete them now. Some snapshots are around 1TB already. The last time we tried to delete a ~500GB snapshot the delete task blocked the whole diskstore's IO and the whole cluster and all hosts became unavailable. Is there a way to delete large snapshots in a humane way that will not block everything? Cheers, Stefan signature.asc Description: OpenPGP digital signature ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
