Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-18 Thread Nir Soffer
On Sun, Feb 18, 2018 at 8:04 PM Alex K  wrote:

> Are there any examples on using ovirt-imageio to backup a VM or where I
> could find details of RESTAPI for this functionality?
> I might attempt to write a python script for this purpose.
>

Here:
-
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/download_disk_snapshots.py
-
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk_snapshots.py

You probably need to add the vm configuration to complete the backup.


>
> Thanx,
> Alex
>
> On Tue, Feb 13, 2018 at 8:59 PM, Alex K  wrote:
>
>> Thank you Nir for the below.
>>
>> I am putting some comments inline in blue.
>>
>>
>> On Tue, Feb 13, 2018 at 7:33 PM, Nir Soffer  wrote:
>>
>>> On Wed, Jan 24, 2018 at 3:19 PM Alex K  wrote:
>>>
 Hi all,

 I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup
 on top glusterfs.
 On some VMs (especially one Windows server 2016 64bit with 500 GB of
 disk). Guest agents are installed at VMs. i almost always observe that
 during the backup of the VM the VM is rendered unresponsive (dashboard
 shows a question mark at the VM status and VM does not respond to ping or
 to anything).

 For scheduled backups I use:

 https://github.com/wefixit-AT/oVirtBackup

 The script does the following:

 1. snapshot VM (this is done ok without any failure)

>>>
>>> This is a very cheap operation
>>>
>>>
 2. Clone snapshot (this steps renders the VM unresponsive)

>>>
>>> This copy 500g of data. In gluster case, it copies 1500g of data, since
>>> in glusterfs, the client
>>> is doing the replication.
>>>
>>> Maybe your network or gluster server is too slow? Can you describe the
>>> network topology?
>>>
>>> Please attach also the volume info for the gluster volume, maybe it is
>>> not configured in the
>>> best way?
>>>
>>
>> The network is 1Gbit. The hosts (3 hosts) are decent ones and new
>> hardware with each host having: 32GB RAM, 16 CPU cores and 2 TB of storage
>> in RAID10.
>> The VMS hosted (7 VMs) exhibit high performance. The VMs are Windows 2016
>> and Windows10.
>> The network topology is: two networks defined at ovirt: ovirtmgmt is for
>> the managment and access network and "storage" is a separate network, where
>> each server is connected with two network cables at a managed switch with
>> mode 6 load balancing. this storage network is used for gluster traffic.
>> Attached the volume configuration.
>>
>>> 3. Export Clone

>>>
>>> This copy 500g to the export domain. If the export domain is on
>>> glusterfs as well, you
>>> copy now another 1500g of data.
>>>
>>>
>> Export domain a Synology NAS with NFS share.  If the cloning succeeds
>> then export is completed ok.
>>
>>> 4. Delete clone

 5. Delete snapshot

>>>
>>> Not clear why do you need to clone the vm before you export it, you can
>>> save half of
>>> the data copies.
>>>
>> Because I cannot export the VM while it is running. It does not provide
>> such option.
>>
>>>
>>> If you 4.2, you can backup the vm *while the vm is running* by:
>>> - Take a snapshot
>>> - Get the vm ovf from the engine api
>>> - Download the vm disks using ovirt-imageio and store the snaphosts in
>>> your backup
>>>   storage
>>> - Delete a snapshot
>>>
>>> In this flow, you would copy 500g.
>>>
>>> I am not aware about this option. checking quickly at site this seems
>> that it is still half implemented? Is there any script that I may use and
>> test this? I am interested to have these backups scheduled.
>>
>>
>>> Daniel, please correct me if I'm wrong regarding doing this online.
>>>
>>> Regardless, a vm should not become non-responsive while cloning. Please
>>> file a bug
>>> for this and attach engine, vdsm, and glusterfs logs.
>>>
>>>
>> Nir
>>>
>>> Do you have any similar experience? Any suggestions to address this?

 I have never seen such issue with hosted Linux VMs.

 The cluster has enough storage to accommodate the clone.


 Thanx,

 Alex



 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users

>>>
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-18 Thread Alex K
Hi all,

Are there any examples on using ovirt-imageio to backup a VM or where I
could find details of RESTAPI for this functionality?
I might attempt to write a python script for this purpose.

Thanx,
Alex

On Tue, Feb 13, 2018 at 8:59 PM, Alex K  wrote:

> Thank you Nir for the below.
>
> I am putting some comments inline in blue.
>
>
> On Tue, Feb 13, 2018 at 7:33 PM, Nir Soffer  wrote:
>
>> On Wed, Jan 24, 2018 at 3:19 PM Alex K  wrote:
>>
>>> Hi all,
>>>
>>> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
>>> top glusterfs.
>>> On some VMs (especially one Windows server 2016 64bit with 500 GB of
>>> disk). Guest agents are installed at VMs. i almost always observe that
>>> during the backup of the VM the VM is rendered unresponsive (dashboard
>>> shows a question mark at the VM status and VM does not respond to ping or
>>> to anything).
>>>
>>> For scheduled backups I use:
>>>
>>> https://github.com/wefixit-AT/oVirtBackup
>>>
>>> The script does the following:
>>>
>>> 1. snapshot VM (this is done ok without any failure)
>>>
>>
>> This is a very cheap operation
>>
>>
>>> 2. Clone snapshot (this steps renders the VM unresponsive)
>>>
>>
>> This copy 500g of data. In gluster case, it copies 1500g of data, since
>> in glusterfs, the client
>> is doing the replication.
>>
>> Maybe your network or gluster server is too slow? Can you describe the
>> network topology?
>>
>> Please attach also the volume info for the gluster volume, maybe it is
>> not configured in the
>> best way?
>>
>
> The network is 1Gbit. The hosts (3 hosts) are decent ones and new hardware
> with each host having: 32GB RAM, 16 CPU cores and 2 TB of storage in
> RAID10.
> The VMS hosted (7 VMs) exhibit high performance. The VMs are Windows 2016
> and Windows10.
> The network topology is: two networks defined at ovirt: ovirtmgmt is for
> the managment and access network and "storage" is a separate network, where
> each server is connected with two network cables at a managed switch with
> mode 6 load balancing. this storage network is used for gluster traffic.
> Attached the volume configuration.
>
>> 3. Export Clone
>>>
>>
>> This copy 500g to the export domain. If the export domain is on glusterfs
>> as well, you
>> copy now another 1500g of data.
>>
>>
> Export domain a Synology NAS with NFS share.  If the cloning succeeds then
> export is completed ok.
>
>> 4. Delete clone
>>>
>>> 5. Delete snapshot
>>>
>>
>> Not clear why do you need to clone the vm before you export it, you can
>> save half of
>> the data copies.
>>
> Because I cannot export the VM while it is running. It does not provide
> such option.
>
>>
>> If you 4.2, you can backup the vm *while the vm is running* by:
>> - Take a snapshot
>> - Get the vm ovf from the engine api
>> - Download the vm disks using ovirt-imageio and store the snaphosts in
>> your backup
>>   storage
>> - Delete a snapshot
>>
>> In this flow, you would copy 500g.
>>
>> I am not aware about this option. checking quickly at site this seems
> that it is still half implemented? Is there any script that I may use and
> test this? I am interested to have these backups scheduled.
>
>
>> Daniel, please correct me if I'm wrong regarding doing this online.
>>
>> Regardless, a vm should not become non-responsive while cloning. Please
>> file a bug
>> for this and attach engine, vdsm, and glusterfs logs.
>>
>>
> Nir
>>
>> Do you have any similar experience? Any suggestions to address this?
>>>
>>> I have never seen such issue with hosted Linux VMs.
>>>
>>> The cluster has enough storage to accommodate the clone.
>>>
>>>
>>> Thanx,
>>>
>>> Alex
>>>
>>>
>>>
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-13 Thread Alex K
Thank you Nir for the below.

I am putting some comments inline in blue.


On Tue, Feb 13, 2018 at 7:33 PM, Nir Soffer  wrote:

> On Wed, Jan 24, 2018 at 3:19 PM Alex K  wrote:
>
>> Hi all,
>>
>> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
>> top glusterfs.
>> On some VMs (especially one Windows server 2016 64bit with 500 GB of
>> disk). Guest agents are installed at VMs. i almost always observe that
>> during the backup of the VM the VM is rendered unresponsive (dashboard
>> shows a question mark at the VM status and VM does not respond to ping or
>> to anything).
>>
>> For scheduled backups I use:
>>
>> https://github.com/wefixit-AT/oVirtBackup
>>
>> The script does the following:
>>
>> 1. snapshot VM (this is done ok without any failure)
>>
>
> This is a very cheap operation
>
>
>> 2. Clone snapshot (this steps renders the VM unresponsive)
>>
>
> This copy 500g of data. In gluster case, it copies 1500g of data, since in
> glusterfs, the client
> is doing the replication.
>
> Maybe your network or gluster server is too slow? Can you describe the
> network topology?
>
> Please attach also the volume info for the gluster volume, maybe it is not
> configured in the
> best way?
>

The network is 1Gbit. The hosts (3 hosts) are decent ones and new hardware
with each host having: 32GB RAM, 16 CPU cores and 2 TB of storage in
RAID10.
The VMS hosted (7 VMs) exhibit high performance. The VMs are Windows 2016
and Windows10.
The network topology is: two networks defined at ovirt: ovirtmgmt is for
the managment and access network and "storage" is a separate network, where
each server is connected with two network cables at a managed switch with
mode 6 load balancing. this storage network is used for gluster traffic.
Attached the volume configuration.

> 3. Export Clone
>>
>
> This copy 500g to the export domain. If the export domain is on glusterfs
> as well, you
> copy now another 1500g of data.
>
>
Export domain a Synology NAS with NFS share.  If the cloning succeeds then
export is completed ok.

> 4. Delete clone
>>
>> 5. Delete snapshot
>>
>
> Not clear why do you need to clone the vm before you export it, you can
> save half of
> the data copies.
>
Because I cannot export the VM while it is running. It does not provide
such option.

>
> If you 4.2, you can backup the vm *while the vm is running* by:
> - Take a snapshot
> - Get the vm ovf from the engine api
> - Download the vm disks using ovirt-imageio and store the snaphosts in
> your backup
>   storage
> - Delete a snapshot
>
> In this flow, you would copy 500g.
>
> I am not aware about this option. checking quickly at site this seems that
it is still half implemented? Is there any script that I may use and test
this? I am interested to have these backups scheduled.


> Daniel, please correct me if I'm wrong regarding doing this online.
>
> Regardless, a vm should not become non-responsive while cloning. Please
> file a bug
> for this and attach engine, vdsm, and glusterfs logs.
>
>
Nir
>
> Do you have any similar experience? Any suggestions to address this?
>>
>> I have never seen such issue with hosted Linux VMs.
>>
>> The cluster has enough storage to accommodate the clone.
>>
>>
>> Thanx,
>>
>> Alex
>>
>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
Volume Name: vms
Type: Replicate
Volume ID: 00fee7f3-76e6-42b2-8f66-606b91df4a97
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster2:/gluster/vms/brick
Brick2: gluster0:/gluster/vms/brick
Brick3: gluster1:/gluster/vms/brick
Options Reconfigured:
features.shard-block-size: 512MB
server.allow-insecure: on
performance.strict-o-direct: on
network.ping-timeout: 30
storage.owner-gid: 36
storage.owner-uid: 36
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 1
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: on
performance.low-prio-threads: 32
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
performance.readdir-ahead: off
nfs.disable: on
nfs.export-volumes: on
cluster.granular-entry-heal: enable
performance.cache-size: 1GB
server.event-threads: 4
client.event-threads: 4
[root@v0 setel]# 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-13 Thread Nir Soffer
On Wed, Jan 24, 2018 at 3:19 PM Alex K  wrote:

> Hi all,
>
> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
> top glusterfs.
> On some VMs (especially one Windows server 2016 64bit with 500 GB of
> disk). Guest agents are installed at VMs. i almost always observe that
> during the backup of the VM the VM is rendered unresponsive (dashboard
> shows a question mark at the VM status and VM does not respond to ping or
> to anything).
>
> For scheduled backups I use:
>
> https://github.com/wefixit-AT/oVirtBackup
>
> The script does the following:
>
> 1. snapshot VM (this is done ok without any failure)
>

This is a very cheap operation


> 2. Clone snapshot (this steps renders the VM unresponsive)
>

This copy 500g of data. In gluster case, it copies 1500g of data, since in
glusterfs, the client
is doing the replication.

Maybe your network or gluster server is too slow? Can you describe the
network topology?

Please attach also the volume info for the gluster volume, maybe it is not
configured in the
best way?


> 3. Export Clone
>

This copy 500g to the export domain. If the export domain is on glusterfs
as well, you
copy now another 1500g of data.


> 4. Delete clone
>
> 5. Delete snapshot
>

Not clear why do you need to clone the vm before you export it, you can
save half of
the data copies.

If you 4.2, you can backup the vm *while the vm is running* by:
- Take a snapshot
- Get the vm ovf from the engine api
- Download the vm disks using ovirt-imageio and store the snaphosts in your
backup
  storage
- Delete a snapshot

In this flow, you would copy 500g.

Daniel, please correct me if I'm wrong regarding doing this online.

Regardless, a vm should not become non-responsive while cloning. Please
file a bug
for this and attach engine, vdsm, and glusterfs logs.

Nir

Do you have any similar experience? Any suggestions to address this?
>
> I have never seen such issue with hosted Linux VMs.
>
> The cluster has enough storage to accommodate the clone.
>
>
> Thanx,
>
> Alex
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-09 Thread Alex K
Hi all,

In case you need any further logs let me know.
Thanx for the time.

Alex

On Thu, Feb 8, 2018 at 9:41 AM, Alex K <rightkickt...@gmail.com> wrote:

> Hi Shani,
>
> Didn't notice that.
> I am attaching later vdsm logs.
>
> Thanx,
> Alex
>
> On Wed, Feb 7, 2018 at 5:31 PM, Shani Leviim <slev...@redhat.com> wrote:
>
>> Hi Alex,
>> Sorry for the mail's delay.
>>
>> From a brief look at your logs, I've noticed that the error you've got at
>> the engine's log was logged at 2018-02-03 00:22:56,
>> while your vdsm's log ends at 2018-02-03 00:01:01.
>> Is there a way you can reproduce a fuller vdsm log?
>>
>>
>> *Regards,*
>>
>> *Shani Leviim*
>>
>> On Sat, Feb 3, 2018 at 5:41 PM, Alex K <rightkickt...@gmail.com> wrote:
>>
>>> Attaching vdm log from host that trigerred the error, where the Vm that
>>> was being cloned was running at that time.
>>>
>>> thanx,
>>> Alex
>>>
>>> On Sat, Feb 3, 2018 at 5:20 PM, Yaniv Kaul <yk...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Feb 3, 2018 3:24 PM, "Alex K" <rightkickt...@gmail.com> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I have reproduced the backups failure. The VM that failed is named
>>>> Win-FileServer and is a Windows 2016 server 64bit with 300GB of disk.
>>>> During the cloning step the VM went unresponsive and I had to
>>>> stop/start it.
>>>> I am attaching the logs.I have another VM with same OS (named DC-Server
>>>> within the logs) but with smaller disk (60GB) which does not give any error
>>>> when it is cloned.
>>>> I see a line:
>>>>
>>>> EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null,
>>>> Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM
>>>> v2.sitedomain command SnapshotVDS failed: Message timeout which can be
>>>> caused by communication issues
>>>>
>>>>
>>>> I suggest adding relevant vdsm.log as well.
>>>> Y.
>>>>
>>>>
>>>> I appreciate any advise why I am facing such issue with the backups.
>>>>
>>>> thanx,
>>>> Alex
>>>>
>>>> On Tue, Jan 30, 2018 at 12:49 AM, Alex K <rightkickt...@gmail.com>
>>>> wrote:
>>>>
>>>>> Ok. I will reproduce and collect logs.
>>>>>
>>>>> Thanx,
>>>>> Alex
>>>>>
>>>>> On Jan 29, 2018 20:21, "Mahdi Adnan" <mahdi.ad...@outlook.com> wrote:
>>>>>
>>>>> I have Windows VMs, both client and server.
>>>>> if you provide the engine.log file we might have a look at it.
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Respectfully
>>>>> *Mahdi A. Mahdi*
>>>>>
>>>>> --
>>>>> *From:* Alex K <rightkickt...@gmail.com>
>>>>> *Sent:* Monday, January 29, 2018 5:40 PM
>>>>> *To:* Mahdi Adnan
>>>>> *Cc:* users
>>>>> *Subject:* Re: [ovirt-users] Ovirt backups lead to unresponsive VM
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have observed this logged at host when the issue occurs:
>>>>>
>>>>> VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer
>>>>>
>>>>> or
>>>>>
>>>>> VDSM host.domain command GetStatsVDS failed: Connection reset by peer
>>>>>
>>>>> At engine logs have not been able to correlate.
>>>>>
>>>>> Are you hosting Windows 2016 server and Windows 10 VMs?
>>>>> The weird is that I have same setup on other clusters with no issues.
>>>>>
>>>>> Thanx,
>>>>> Alex
>>>>>
>>>>> On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan <mahdi.ad...@outlook.com>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have a cluster of 17 nodes, backed by GlusterFS storage, and using
>>>>> this same script for backup.
>>>>> we have no issues with it so far.
>>>>> have you checked engine log file ?
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Respectfully
>>>&g

Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-07 Thread Shani Leviim
Hi Alex,
Sorry for the mail's delay.

>From a brief look at your logs, I've noticed that the error you've got at
the engine's log was logged at 2018-02-03 00:22:56,
while your vdsm's log ends at 2018-02-03 00:01:01.
Is there a way you can reproduce a fuller vdsm log?


*Regards,*

*Shani Leviim*

On Sat, Feb 3, 2018 at 5:41 PM, Alex K <rightkickt...@gmail.com> wrote:

> Attaching vdm log from host that trigerred the error, where the Vm that
> was being cloned was running at that time.
>
> thanx,
> Alex
>
> On Sat, Feb 3, 2018 at 5:20 PM, Yaniv Kaul <yk...@redhat.com> wrote:
>
>>
>>
>> On Feb 3, 2018 3:24 PM, "Alex K" <rightkickt...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I have reproduced the backups failure. The VM that failed is named
>> Win-FileServer and is a Windows 2016 server 64bit with 300GB of disk.
>> During the cloning step the VM went unresponsive and I had to stop/start
>> it.
>> I am attaching the logs.I have another VM with same OS (named DC-Server
>> within the logs) but with smaller disk (60GB) which does not give any error
>> when it is cloned.
>> I see a line:
>>
>> EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call
>> Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM
>> v2.sitedomain command SnapshotVDS failed: Message timeout which can be
>> caused by communication issues
>>
>>
>> I suggest adding relevant vdsm.log as well.
>> Y.
>>
>>
>> I appreciate any advise why I am facing such issue with the backups.
>>
>> thanx,
>> Alex
>>
>> On Tue, Jan 30, 2018 at 12:49 AM, Alex K <rightkickt...@gmail.com> wrote:
>>
>>> Ok. I will reproduce and collect logs.
>>>
>>> Thanx,
>>> Alex
>>>
>>> On Jan 29, 2018 20:21, "Mahdi Adnan" <mahdi.ad...@outlook.com> wrote:
>>>
>>> I have Windows VMs, both client and server.
>>> if you provide the engine.log file we might have a look at it.
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* Alex K <rightkickt...@gmail.com>
>>> *Sent:* Monday, January 29, 2018 5:40 PM
>>> *To:* Mahdi Adnan
>>> *Cc:* users
>>> *Subject:* Re: [ovirt-users] Ovirt backups lead to unresponsive VM
>>>
>>> Hi,
>>>
>>> I have observed this logged at host when the issue occurs:
>>>
>>> VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer
>>>
>>> or
>>>
>>> VDSM host.domain command GetStatsVDS failed: Connection reset by peer
>>>
>>> At engine logs have not been able to correlate.
>>>
>>> Are you hosting Windows 2016 server and Windows 10 VMs?
>>> The weird is that I have same setup on other clusters with no issues.
>>>
>>> Thanx,
>>> Alex
>>>
>>> On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan <mahdi.ad...@outlook.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> We have a cluster of 17 nodes, backed by GlusterFS storage, and using
>>> this same script for backup.
>>> we have no issues with it so far.
>>> have you checked engine log file ?
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>> --
>>> *From:* users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of
>>> Alex K <rightkickt...@gmail.com>
>>> *Sent:* Wednesday, January 24, 2018 4:18 PM
>>> *To:* users
>>> *Subject:* [ovirt-users] Ovirt backups lead to unresponsive VM
>>>
>>> Hi all,
>>>
>>> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
>>> top glusterfs.
>>> On some VMs (especially one Windows server 2016 64bit with 500 GB of
>>> disk). Guest agents are installed at VMs. i almost always observe that
>>> during the backup of the VM the VM is rendered unresponsive (dashboard
>>> shows a question mark at the VM status and VM does not respond to ping or
>>> to anything).
>>>
>>> For scheduled backups I use:
>>>
>>> https://github.com/wefixit-AT/oVirtBackup
>>>
>>> The script does the following:
>>>
>>> 1. snapshot VM (this is done ok without any failure)
>>>
>>> 2. Clone snapshot (this steps renders the VM unresponsive)
>>>
>>> 3. Export Clone
>>>
>>> 4. Delete clone
>>>
>>> 5. Delete snapshot
>>>
>>>
>>> Do you have any similar experience? Any suggestions to address this?
>>>
>>> I have never seen such issue with hosted Linux VMs.
>>>
>>> The cluster has enough storage to accommodate the clone.
>>>
>>>
>>> Thanx,
>>>
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-02-03 Thread Yaniv Kaul
On Feb 3, 2018 3:24 PM, "Alex K" <rightkickt...@gmail.com> wrote:

Hi All,

I have reproduced the backups failure. The VM that failed is named
Win-FileServer and is a Windows 2016 server 64bit with 300GB of disk.
During the cloning step the VM went unresponsive and I had to stop/start
it.
I am attaching the logs.I have another VM with same OS (named DC-Server
within the logs) but with smaller disk (60GB) which does not give any error
when it is cloned.
I see a line:

EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call
Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM
v2.sitedomain command SnapshotVDS failed: Message timeout which can be
caused by communication issues


I suggest adding relevant vdsm.log as well.
Y.


I appreciate any advise why I am facing such issue with the backups.

thanx,
Alex

On Tue, Jan 30, 2018 at 12:49 AM, Alex K <rightkickt...@gmail.com> wrote:

> Ok. I will reproduce and collect logs.
>
> Thanx,
> Alex
>
> On Jan 29, 2018 20:21, "Mahdi Adnan" <mahdi.ad...@outlook.com> wrote:
>
> I have Windows VMs, both client and server.
> if you provide the engine.log file we might have a look at it.
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* Alex K <rightkickt...@gmail.com>
> *Sent:* Monday, January 29, 2018 5:40 PM
> *To:* Mahdi Adnan
> *Cc:* users
> *Subject:* Re: [ovirt-users] Ovirt backups lead to unresponsive VM
>
> Hi,
>
> I have observed this logged at host when the issue occurs:
>
> VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer
>
> or
>
> VDSM host.domain command GetStatsVDS failed: Connection reset by peer
>
> At engine logs have not been able to correlate.
>
> Are you hosting Windows 2016 server and Windows 10 VMs?
> The weird is that I have same setup on other clusters with no issues.
>
> Thanx,
> Alex
>
> On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan <mahdi.ad...@outlook.com>
> wrote:
>
> Hi,
>
> We have a cluster of 17 nodes, backed by GlusterFS storage, and using this
> same script for backup.
> we have no issues with it so far.
> have you checked engine log file ?
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ----------
> *From:* users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of
> Alex K <rightkickt...@gmail.com>
> *Sent:* Wednesday, January 24, 2018 4:18 PM
> *To:* users
> *Subject:* [ovirt-users] Ovirt backups lead to unresponsive VM
>
> Hi all,
>
> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
> top glusterfs.
> On some VMs (especially one Windows server 2016 64bit with 500 GB of
> disk). Guest agents are installed at VMs. i almost always observe that
> during the backup of the VM the VM is rendered unresponsive (dashboard
> shows a question mark at the VM status and VM does not respond to ping or
> to anything).
>
> For scheduled backups I use:
>
> https://github.com/wefixit-AT/oVirtBackup
>
> The script does the following:
>
> 1. snapshot VM (this is done ok without any failure)
>
> 2. Clone snapshot (this steps renders the VM unresponsive)
>
> 3. Export Clone
>
> 4. Delete clone
>
> 5. Delete snapshot
>
>
> Do you have any similar experience? Any suggestions to address this?
>
> I have never seen such issue with hosted Linux VMs.
>
> The cluster has enough storage to accommodate the clone.
>
>
> Thanx,
>
> Alex
>
>
>
>
>
>

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-01-30 Thread Mahdi Adnan
I have Windows VMs, both client and server.
if you provide the engine.log file we might have a look at it.


--

Respectfully
Mahdi A. Mahdi


From: Alex K <rightkickt...@gmail.com>
Sent: Monday, January 29, 2018 5:40 PM
To: Mahdi Adnan
Cc: users
Subject: Re: [ovirt-users] Ovirt backups lead to unresponsive VM

Hi,

I have observed this logged at host when the issue occurs:

VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer

or

VDSM host.domain command GetStatsVDS failed: Connection reset by peer

At engine logs have not been able to correlate.

Are you hosting Windows 2016 server and Windows 10 VMs?
The weird is that I have same setup on other clusters with no issues.

Thanx,
Alex

On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan 
<mahdi.ad...@outlook.com<mailto:mahdi.ad...@outlook.com>> wrote:
Hi,

We have a cluster of 17 nodes, backed by GlusterFS storage, and using this same 
script for backup.
we have no issues with it so far.
have you checked engine log file ?


--

Respectfully
Mahdi A. Mahdi


From: users-boun...@ovirt.org<mailto:users-boun...@ovirt.org> 
<users-boun...@ovirt.org<mailto:users-boun...@ovirt.org>> on behalf of Alex K 
<rightkickt...@gmail.com<mailto:rightkickt...@gmail.com>>
Sent: Wednesday, January 24, 2018 4:18 PM
To: users
Subject: [ovirt-users] Ovirt backups lead to unresponsive VM

Hi all,

I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on top 
glusterfs.
On some VMs (especially one Windows server 2016 64bit with 500 GB of disk). 
Guest agents are installed at VMs. i almost always observe that during the 
backup of the VM the VM is rendered unresponsive (dashboard shows a question 
mark at the VM status and VM does not respond to ping or to anything).

For scheduled backups I use:

https://github.com/wefixit-AT/oVirtBackup

The script does the following:

1. snapshot VM (this is done ok without any failure)

2. Clone snapshot (this steps renders the VM unresponsive)

3. Export Clone

4. Delete clone

5. Delete snapshot


Do you have any similar experience? Any suggestions to address this?

I have never seen such issue with hosted Linux VMs.

The cluster has enough storage to accommodate the clone.


Thanx,

Alex



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-01-29 Thread Alex K
Ok. I will reproduce and collect logs.

Thanx,
Alex

On Jan 29, 2018 20:21, "Mahdi Adnan" <mahdi.ad...@outlook.com> wrote:

I have Windows VMs, both client and server.
if you provide the engine.log file we might have a look at it.


-- 

Respectfully
*Mahdi A. Mahdi*

--
*From:* Alex K <rightkickt...@gmail.com>
*Sent:* Monday, January 29, 2018 5:40 PM
*To:* Mahdi Adnan
*Cc:* users
*Subject:* Re: [ovirt-users] Ovirt backups lead to unresponsive VM

Hi,

I have observed this logged at host when the issue occurs:

VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer

or

VDSM host.domain command GetStatsVDS failed: Connection reset by peer

At engine logs have not been able to correlate.

Are you hosting Windows 2016 server and Windows 10 VMs?
The weird is that I have same setup on other clusters with no issues.

Thanx,
Alex

On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan <mahdi.ad...@outlook.com>
wrote:

Hi,

We have a cluster of 17 nodes, backed by GlusterFS storage, and using this
same script for backup.
we have no issues with it so far.
have you checked engine log file ?


-- 

Respectfully
*Mahdi A. Mahdi*

--
*From:* users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of Alex
K <rightkickt...@gmail.com>
*Sent:* Wednesday, January 24, 2018 4:18 PM
*To:* users
*Subject:* [ovirt-users] Ovirt backups lead to unresponsive VM

Hi all,

I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
top glusterfs.
On some VMs (especially one Windows server 2016 64bit with 500 GB of disk).
Guest agents are installed at VMs. i almost always observe that during the
backup of the VM the VM is rendered unresponsive (dashboard shows a
question mark at the VM status and VM does not respond to ping or to
anything).

For scheduled backups I use:

https://github.com/wefixit-AT/oVirtBackup

The script does the following:

1. snapshot VM (this is done ok without any failure)

2. Clone snapshot (this steps renders the VM unresponsive)

3. Export Clone

4. Delete clone

5. Delete snapshot


Do you have any similar experience? Any suggestions to address this?

I have never seen such issue with hosted Linux VMs.

The cluster has enough storage to accommodate the clone.


Thanx,

Alex
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-01-29 Thread Alex K
Hi,

I have observed this logged at host when the issue occurs:

VDSM command GetStoragePoolInfoVDS failed: Connection reset by peer

or

VDSM host.domain command GetStatsVDS failed: Connection reset by peer

At engine logs have not been able to correlate.

Are you hosting Windows 2016 server and Windows 10 VMs?
The weird is that I have same setup on other clusters with no issues.

Thanx,
Alex

On Sun, Jan 28, 2018 at 9:21 PM, Mahdi Adnan <mahdi.ad...@outlook.com>
wrote:

> Hi,
>
> We have a cluster of 17 nodes, backed by GlusterFS storage, and using this
> same script for backup.
> we have no issues with it so far.
> have you checked engine log file ?
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> --
> *From:* users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of
> Alex K <rightkickt...@gmail.com>
> *Sent:* Wednesday, January 24, 2018 4:18 PM
> *To:* users
> *Subject:* [ovirt-users] Ovirt backups lead to unresponsive VM
>
> Hi all,
>
> I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
> top glusterfs.
> On some VMs (especially one Windows server 2016 64bit with 500 GB of
> disk). Guest agents are installed at VMs. i almost always observe that
> during the backup of the VM the VM is rendered unresponsive (dashboard
> shows a question mark at the VM status and VM does not respond to ping or
> to anything).
>
> For scheduled backups I use:
>
> https://github.com/wefixit-AT/oVirtBackup
>
> The script does the following:
>
> 1. snapshot VM (this is done ok without any failure)
>
> 2. Clone snapshot (this steps renders the VM unresponsive)
>
> 3. Export Clone
>
> 4. Delete clone
>
> 5. Delete snapshot
>
>
> Do you have any similar experience? Any suggestions to address this?
>
> I have never seen such issue with hosted Linux VMs.
>
> The cluster has enough storage to accommodate the clone.
>
>
> Thanx,
>
> Alex
>
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] Ovirt backups lead to unresponsive VM

2018-01-28 Thread Mahdi Adnan
Hi,

We have a cluster of 17 nodes, backed by GlusterFS storage, and using this same 
script for backup.
we have no issues with it so far.
have you checked engine log file ?


--

Respectfully
Mahdi A. Mahdi


From: users-boun...@ovirt.org <users-boun...@ovirt.org> on behalf of Alex K 
<rightkickt...@gmail.com>
Sent: Wednesday, January 24, 2018 4:18 PM
To: users
Subject: [ovirt-users] Ovirt backups lead to unresponsive VM

Hi all,

I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on top 
glusterfs.
On some VMs (especially one Windows server 2016 64bit with 500 GB of disk). 
Guest agents are installed at VMs. i almost always observe that during the 
backup of the VM the VM is rendered unresponsive (dashboard shows a question 
mark at the VM status and VM does not respond to ping or to anything).

For scheduled backups I use:

https://github.com/wefixit-AT/oVirtBackup

The script does the following:

1. snapshot VM (this is done ok without any failure)

2. Clone snapshot (this steps renders the VM unresponsive)

3. Export Clone

4. Delete clone

5. Delete snapshot


Do you have any similar experience? Any suggestions to address this?

I have never seen such issue with hosted Linux VMs.

The cluster has enough storage to accommodate the clone.


Thanx,

Alex


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


[ovirt-users] Ovirt backups lead to unresponsive VM

2018-01-24 Thread Alex K
Hi all,

I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on
top glusterfs.
On some VMs (especially one Windows server 2016 64bit with 500 GB of disk).
Guest agents are installed at VMs. i almost always observe that during the
backup of the VM the VM is rendered unresponsive (dashboard shows a
question mark at the VM status and VM does not respond to ping or to
anything).

For scheduled backups I use:

https://github.com/wefixit-AT/oVirtBackup

The script does the following:

1. snapshot VM (this is done ok without any failure)

2. Clone snapshot (this steps renders the VM unresponsive)

3. Export Clone

4. Delete clone

5. Delete snapshot


Do you have any similar experience? Any suggestions to address this?

I have never seen such issue with hosted Linux VMs.

The cluster has enough storage to accommodate the clone.


Thanx,

Alex
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users