Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Pavel Gashev]

It really works. The bug is that engine doesn't update the state of 
finalization step. Engine.log shows that finalization is actually finished. 
Restart of engine fixes it. Also it's possible to fix it in directly in 
database w/o restarting anything.


From: Markus Stockhausen <stockhau...@collogia.de>
Sent: Apr 13, 2016 10:23 PM
To: Gianluca Cecchi
Cc: Pavel Gashev; users
Subject: AW: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

Hi,

I was just annoyed that during my tests I got a second error. For me
it seemed as finally everything worked fine. Although the single disk live
merge task was shown to be still running finally.

Nevertheless after Nirs explanation for BZ1326900 I'm unsure if the single
disk live merge really works. So we currently have the following situation:

Live merge (for all disks) stalls the VM

Live merge (for single disks) seems to work but logs give other info.

Markus


Von: Gianluca Cecchi [gianluca.cec...@gmail.com]
Gesendet: Mittwoch, 13. April 2016 20:59
An: Markus Stockhausen
Cc: Pavel Gashev; users
Betreff: Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

On Wed, Apr 13, 2016 at 7:54 PM, Markus Stockhausen 
<stockhau...@collogia.de<mailto:stockhau...@collogia.de>> wrote:
Hi Pavel,

I don't understand...


I tested as advised. And aaargh hit the next two OVirt bugs

https://bugzilla.redhat.com/show_bug.cgi?id=1326900

>From above it seems that also one by one doesn't work for you, and going to 
>bugzilla link seems to confirm that.



Nevertheless: Merging only one disk at once works as expected.
The VM stays up, keeps running and is responsive.

Markus


But here above you are saying instead that it works as expected...?


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Nir Soffer]

On Wed, Apr 13, 2016 at 10:23 PM, Markus Stockhausen
<stockhau...@collogia.de> wrote:
> Hi,
>
> I was just annoyed that during my tests I got a second error. For me
> it seemed as finally everything worked fine. Although the single disk live
> merge task was shown to be still running finally.
>
> Nevertheless after Nirs explanation for BZ1326900 I'm unsure if the single
> disk live merge really works. So we currently have the following situation:
>
> Live merge (for all disks) stalls the VM
>
> Live merge (for single disks) seems to work but logs give other info.

Deeper look into vdsm log show that the merge was successful, so we
have issue on the engine side, not displaying the state correctly.

Nir

>
> Markus
>
>
> Von: Gianluca Cecchi [gianluca.cec...@gmail.com]
> Gesendet: Mittwoch, 13. April 2016 20:59
> An: Markus Stockhausen
> Cc: Pavel Gashev; users
> Betreff: Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3
>
> On Wed, Apr 13, 2016 at 7:54 PM, Markus Stockhausen
> <stockhau...@collogia.de> wrote:
>>
>> Hi Pavel,
>
>
> I don't understand...
>
>>
>>
>> I tested as advised. And aaargh hit the next two OVirt bugs
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1326900
>
>
> From above it seems that also one by one doesn't work for you, and going to
> bugzilla link seems to confirm that.
>
>
>>
>>
>> Nevertheless: Merging only one disk at once works as expected.
>> The VM stays up, keeps running and is responsive.
>>
>> Markus
>> 
>
>
> But here above you are saying instead that it works as expected...?
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Markus Stockhausen]

Hi,

I was just annoyed that during my tests I got a second error. For me
it seemed as finally everything worked fine. Although the single disk live
merge task was shown to be still running finally.

Nevertheless after Nirs explanation for BZ1326900 I'm unsure if the single
disk live merge really works. So we currently have the following situation:

Live merge (for all disks) stalls the VM

Live merge (for single disks) seems to work but logs give other info.

Markus


Von: Gianluca Cecchi [gianluca.cec...@gmail.com]
Gesendet: Mittwoch, 13. April 2016 20:59
An: Markus Stockhausen
Cc: Pavel Gashev; users
Betreff: Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

On Wed, Apr 13, 2016 at 7:54 PM, Markus Stockhausen 
<stockhau...@collogia.de<mailto:stockhau...@collogia.de>> wrote:
Hi Pavel,

I don't understand...


I tested as advised. And aaargh hit the next two OVirt bugs

https://bugzilla.redhat.com/show_bug.cgi?id=1326900

>From above it seems that also one by one doesn't work for you, and going to 
>bugzilla link seems to confirm that.



Nevertheless: Merging only one disk at once works as expected.
The VM stays up, keeps running and is responsive.

Markus


But here above you are saying instead that it works as expected...?



Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Gianluca Cecchi]

On Wed, Apr 13, 2016 at 7:54 PM, Markus Stockhausen  wrote:

> Hi Pavel,
>

I don't understand...


>
> I tested as advised. And aaargh hit the next two OVirt bugs
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1326900
>

>From above it seems that also one by one doesn't work for you, and going to
bugzilla link seems to confirm that.



>
> Nevertheless: Merging only one disk at once works as expected.
> The VM stays up, keeps running and is responsive.
>
> Markus
> 
>

But here above you are saying instead that it works as expected...?
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Markus Stockhausen]

Hi Pavel,

I tested as advised. And aaargh hit the next two OVirt bugs

https://bugzilla.redhat.com/show_bug.cgi?id=1326900

Nevertheless: Merging only one disk at once works as expected.
The VM stays up, keeps running and is responsive.

Markus

Von: Pavel Gashev [p...@acronis.com]
Gesendet: Mittwoch, 13. April 2016 15:12
An: Markus Stockhausen; users
Betreff: Re: AW: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

Markus,

So all CPU threads are blocked by the main loop. The main loop is busy draining 
IO requests from all drives while there are other block commit jobs active... 
It looks like currently QEMU just can't live merge more than one drive 
simultaneously without blocking VCPUs.

As workaround you can delete disk snapshots one-by-one, and then delete 
disk-less VM snapshot. The Storage tab has Disk Snapshots sub tab where you can 
remove a specific snapshot of a specific disk.





On 12/04/16 21:39, "Markus Stockhausen" <stockhau...@collogia.de> wrote:

>> Von: Pavel Gashev [p...@acronis.com]
>> Gesendet: Dienstag, 12. April 2016 16:15
>> An: Markus Stockhausen; users
>> Betreff: Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3
>>
>> Markus,
>>
>> I saw similar issues. Looks like it's related to multidisk VMs under high 
>> disk load. No relations with guest OS. No relations with IO threads.
>>
>>
>> I didn't debug it. Can you please show a backtrace of a thread in 
>> __lll_lock_wait ?
>
>Hi Pavel.
>
>thanks for your response. I dont even need high disk load to
>reproduce the issue. Just a simple script that writes 8k blocks
>every 0.5 secs.  I added the required info to the BZ.
>
>https://bugzilla.redhat.com/show_bug.cgi?id=1319400
>
>Markus
>
>>
>>
>> On 11/04/16 18:13, "users-boun...@ovirt.org on behalf of Markus Stockhausen" 
>> <users-boun...@ovirt.org on behalf of stockhau...@collogia.de> wrote:
>>
>> >Hi there,
>> >
>> >I'm getting slowly mad about our new Centos 7 cluster. Whenever
>> >I start a live merge the machine completely freezes. It seems to be
>> >independent of the Guest OS (tried SLES 11 SP3, SLES11 SP4 and
>> >SLES12). I already opened BZ1319400 because I'm clueless.
>> >
>> >Doing the same in our Fedora 20 cluster (qemu 2.1.3) we only see
>> >small hiccups of 1-3 seconds at the start of the live merge but no
>> >complete stall. Its the same VM!
>> >
>> >Just to be sure that it is not a specific problem of our topology
>> >or SLES I would like to receive some feedback of the list if anyone
>> >observes similar lockups in their environment.
>> >
>> >Thanks in advance.
>> >
>> >Markus
>> >

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

[Markus Stockhausen]

Ok will give it a try...

Von: Pavel Gashev [p...@acronis.com]
Gesendet: Mittwoch, 13. April 2016 15:12
An: Markus Stockhausen; users
Betreff: Re: AW: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3

Markus,

So all CPU threads are blocked by the main loop. The main loop is busy draining 
IO requests from all drives while there are other block commit jobs active... 
It looks like currently QEMU just can't live merge more than one drive 
simultaneously without blocking VCPUs.

As workaround you can delete disk snapshots one-by-one, and then delete 
disk-less VM snapshot. The Storage tab has Disk Snapshots sub tab where you can 
remove a specific snapshot of a specific disk.





On 12/04/16 21:39, "Markus Stockhausen" <stockhau...@collogia.de> wrote:

>> Von: Pavel Gashev [p...@acronis.com]
>> Gesendet: Dienstag, 12. April 2016 16:15
>> An: Markus Stockhausen; users
>> Betreff: Re: [ovirt-users] stalls during live Merge Centos 7 / qemu 2.3
>>
>> Markus,
>>
>> I saw similar issues. Looks like it's related to multidisk VMs under high 
>> disk load. No relations with guest OS. No relations with IO threads.
>>
>>
>> I didn't debug it. Can you please show a backtrace of a thread in 
>> __lll_lock_wait ?
>
>Hi Pavel.
>
>thanks for your response. I dont even need high disk load to
>reproduce the issue. Just a simple script that writes 8k blocks
>every 0.5 secs.  I added the required info to the BZ.
>
>https://bugzilla.redhat.com/show_bug.cgi?id=1319400
>
>Markus
>
>>
>>
>> On 11/04/16 18:13, "users-boun...@ovirt.org on behalf of Markus Stockhausen" 
>> <users-boun...@ovirt.org on behalf of stockhau...@collogia.de> wrote:
>>
>> >Hi there,
>> >
>> >I'm getting slowly mad about our new Centos 7 cluster. Whenever
>> >I start a live merge the machine completely freezes. It seems to be
>> >independent of the Guest OS (tried SLES 11 SP3, SLES11 SP4 and
>> >SLES12). I already opened BZ1319400 because I'm clueless.
>> >
>> >Doing the same in our Fedora 20 cluster (qemu 2.1.3) we only see
>> >small hiccups of 1-3 seconds at the start of the live merge but no
>> >complete stall. Its the same VM!
>> >
>> >Just to be sure that it is not a specific problem of our topology
>> >or SLES I would like to receive some feedback of the list if anyone
>> >observes similar lockups in their environment.
>> >
>> >Thanks in advance.
>> >
>> >Markus
>> >

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users