Re: [ovirt-users] trouble when creating VM snapshots including memory

[Matthias Leopold]

Am 2017-06-11 um 10:11 schrieb Yaniv Kaul:



On Fri, Jun 9, 2017 at 3:39 PM, Matthias Leopold 
> wrote:


hi,

i'm having trouble creating VM snapshots that include memory in my
oVirt 4.1 test environment. when i do this the VM gets paused and
shortly (20-30s) afterwards i'm seeing messages in engine.log about
both iSCSI storage domains (master storage domain and data storage
where VM resides) experiencing high latency. this quickly worsens
from the engines view: VM is unresponsive, Host is unresponsive,
engine wants to fence the host (impossible because it's the only
host in the test cluster). in the end there is an EngineException

EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can
be caused by communication issues (Failed with error
VDS_NETWORK_ERROR and code 5022)

the snapshot fails and is left in an inconsistent state. the
situation has to be resolved manually with unlock_entity.sh and
maybe lvm commands. this happened twice in exactly the same manner.
VM snapshots without memory for this VM are not a problem.

VM guest OS is CentOS7 installed from one of the
ovirt-image-repository images. it has the oVirt guest agent running.

what could be wrong?

this is a test environment where lots of parameters aren't optimal
but i never had problems like this before, nothing concerning
network latency. iSCSI is on a FreeNAS box. CPU, RAM, ethernet
(10GBit for storage) on all hosts involved (engine hosted
externally, oVirt Node, storage) should be OK by far.


Are you sure iSCSI traffic is going over the 10gb interfaces?
If it doesn't, it might choke the mgmt interface.
Regardless, how is the performance of the storage? I don't expect it to 
require too much, but saving the memory might require some storage 
performance. Perhaps there's a bottleneck there?

Y.


i shot myself in the foot by also playing around with network QoS and 
forgetting about it no wonder the network chokes when i tell it to 
do so. without randomly applied QoS profiles snapshots work perfectly ;-)


thx
matthias



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] trouble when creating VM snapshots including memory

[Matthias Leopold]

Am 2017-06-09 um 21:48 schrieb Karli Sjöberg:



Den 9 juni 2017 21:40 skrev Matthias Leopold 
:


hi,

i'm having trouble creating VM snapshots that include memory in my
oVirt
4.1 test environment. when i do this the VM gets paused and shortly
(20-30s) afterwards i'm seeing messages in engine.log about both iSCSI
storage domains (master storage domain and data storage where VM
resides) experiencing high latency. this quickly worsens from the
engines view: VM is unresponsive, Host is unresponsive, engine wants to
fence the host (impossible because it's the only host in the test
cluster). in the end there is an EngineException

EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)

the snapshot fails and is left in an inconsistent state. the situation
has to be resolved manually with unlock_entity.sh and maybe lvm
commands. this happened twice in exactly the same manner. VM snapshots
without memory for this VM are not a problem.

VM guest OS is CentOS7 installed from one of the ovirt-image-repository
images. it has the oVirt guest agent running.

what could be wrong?


Seems to me that the snapshot operation, where the host needs to save 
all of the VM memory chokes the storage pipe, the host becomes 
"unresponsive" from engine's point of view and all goes up shit creek. 
How is the hypervisor connected to the storage, in more detail?


/K


i shot myself in the foot by also playing around with network QoS and 
forgetting about it no wonder the network chokes when i tell it to 
do so. without randomly applied QoS profiles snapshots work perfectly ;-)


thx
matthias


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] trouble when creating VM snapshots including memory

[Yaniv Kaul]

On Fri, Jun 9, 2017 at 3:39 PM, Matthias Leopold <
matthias.leop...@meduniwien.ac.at> wrote:

> hi,
>
> i'm having trouble creating VM snapshots that include memory in my oVirt
> 4.1 test environment. when i do this the VM gets paused and shortly
> (20-30s) afterwards i'm seeing messages in engine.log about both iSCSI
> storage domains (master storage domain and data storage where VM resides)
> experiencing high latency. this quickly worsens from the engines view: VM
> is unresponsive, Host is unresponsive, engine wants to fence the host
> (impossible because it's the only host in the test cluster). in the end
> there is an EngineException
>
> EngineException: org.ovirt.engine.core.vdsbroke
> r.vdsbroker.VDSNetworkException: VDSGenericException:
> VDSNetworkException: Message timeout which can be caused by communication
> issues (Failed with error VDS_NETWORK_ERROR and code 5022)
>
> the snapshot fails and is left in an inconsistent state. the situation has
> to be resolved manually with unlock_entity.sh and maybe lvm commands. this
> happened twice in exactly the same manner. VM snapshots without memory for
> this VM are not a problem.
>
> VM guest OS is CentOS7 installed from one of the ovirt-image-repository
> images. it has the oVirt guest agent running.
>
> what could be wrong?
>
> this is a test environment where lots of parameters aren't optimal but i
> never had problems like this before, nothing concerning network latency.
> iSCSI is on a FreeNAS box. CPU, RAM, ethernet (10GBit for storage) on all
> hosts involved (engine hosted externally, oVirt Node, storage) should be OK
> by far.
>

Are you sure iSCSI traffic is going over the 10gb interfaces?
If it doesn't, it might choke the mgmt interface.
Regardless, how is the performance of the storage? I don't expect it to
require too much, but saving the memory might require some storage
performance. Perhaps there's a bottleneck there?
Y.


>
> it looks like some obvious configuration botch or performance bottleneck
> to me. can it be linked to the network roles (management and migration
> network are on a 1 GBit link)?
>
> i'm still new to this, not a lot of KVM experience, too. maybe someone
> recognizes the culprit...
>
> thx
> matthias
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] trouble when creating VM snapshots including memory

[Karli Sjöberg]

Den 9 juni 2017 21:40 skrev Matthias Leopold :hi,i'm having trouble creating VM snapshots that include memory in my oVirt 4.1 test environment. when i do this the VM gets paused and shortly (20-30s) afterwards i'm seeing messages in engine.log about both iSCSI storage domains (master storage domain and data storage where VM resides) experiencing high latency. this quickly worsens from the engines view: VM is unresponsive, Host is unresponsive, engine wants to fence the host (impossible because it's the only host in the test cluster). in the end there is an EngineExceptionEngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues (Failed with error VDS_NETWORK_ERROR and code 5022)the snapshot fails and is left in an inconsistent state. the situation has to be resolved manually with unlock_entity.sh and maybe lvm commands. this happened twice in exactly the same manner. VM snapshots without memory for this VM are not a problem.VM guest OS is CentOS7 installed from one of the ovirt-image-repository images. it has the oVirt guest agent running.what could be wrong?Seems to me that the snapshot operation, where the host needs to save all of the VM memory chokes the storage pipe, the host becomes "unresponsive" from engine's point of view and all goes up shit creek. How is the hypervisor connected to the storage, in more detail?/Kthis is a test environment where lots of parameters aren't optimal but i never had problems like this before, nothing concerning network latency. iSCSI is on a FreeNAS box. CPU, RAM, ethernet (10GBit for storage) on all hosts involved (engine hosted externally, oVirt Node, storage) should be OK by far.it looks like some obvious configuration botch or performance bottleneck to me. can it be linked to the network roles (management and migration network are on a 1 GBit link)?i'm still new to this, not a lot of KVM experience, too. maybe someone recognizes the culprit...thxmatthias___Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] trouble when creating VM snapshots including memory

[Matthias Leopold]

hi,

i'm having trouble creating VM snapshots that include memory in my oVirt 
4.1 test environment. when i do this the VM gets paused and shortly 
(20-30s) afterwards i'm seeing messages in engine.log about both iSCSI 
storage domains (master storage domain and data storage where VM 
resides) experiencing high latency. this quickly worsens from the 
engines view: VM is unresponsive, Host is unresponsive, engine wants to 
fence the host (impossible because it's the only host in the test 
cluster). in the end there is an EngineException


EngineException: 
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: 
VDSGenericException: VDSNetworkException: Message timeout which can be 
caused by communication issues (Failed with error VDS_NETWORK_ERROR and 
code 5022)


the snapshot fails and is left in an inconsistent state. the situation 
has to be resolved manually with unlock_entity.sh and maybe lvm 
commands. this happened twice in exactly the same manner. VM snapshots 
without memory for this VM are not a problem.


VM guest OS is CentOS7 installed from one of the ovirt-image-repository 
images. it has the oVirt guest agent running.


what could be wrong?

this is a test environment where lots of parameters aren't optimal but i 
never had problems like this before, nothing concerning network latency. 
iSCSI is on a FreeNAS box. CPU, RAM, ethernet (10GBit for storage) on 
all hosts involved (engine hosted externally, oVirt Node, storage) 
should be OK by far.


it looks like some obvious configuration botch or performance bottleneck 
to me. can it be linked to the network roles (management and migration 
network are on a 1 GBit link)?


i'm still new to this, not a lot of KVM experience, too. maybe someone 
recognizes the culprit...


thx
matthias
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users