Re: [ovirt-users] Ovirt causing strange network issues?

[Colin Coe]

Hi

We saw something very similar to this a couple of years ago.  In our case,
it was caused by STP being enabled on our hypervisors.

HTH



On 3 Oct. 2017 04:56, "Derek Atkins"  wrote:

> Hi,
>
> I'm at my wits end so I'm tossing this here in the hopes that SOMEONE
> will be able to help me.
>
> tl;dr: Ovirt is doing something on my network that is causing my fiber
> modem to go from 3-5ms to 300-1000+ms round trip times.  I know it's
> ovirt because when I unplug ovirt from my network the issue goes away;
> when I plug it back in, the issue recurs.
>
> Long version:
>
> I've been running Ovirt 4.0.6 happily on CentOS 7.3 for several months
> on a single host machine. Indeed, the host had an uptime of 200+ days
> and was working great until approximately midnight, September 21/22
> (just over a week ago).  I was on an airplane halfway across the
> Atlantic at that time, so it wasn't anything I did.
>
> My network is configured as:
>
>   fiber modem <-> edgerouter <-> switch <-> everything else
>
> ovirt is living in the "everything else" area.
>
> When I sit with a laptop connected to either the everything else range
> or even directly connected to the fiber modem, I run 'mtr' and see
> network times (starting at the fiber modem) that bounce all over the
> place.  When I unplug ovirt I see consistent 3-5ms times.  Plug it back
> in, voom, back up to badness.
>
> I've spent several hours plugging and unplugging different devices
> trying to isolate the issue.  The only "device" that has any effect is
> my ovirt box.
>
> I have tried to debug this in several ways, but really the only thing
> that seems to have helped at all is shutting down all the VMs and the
> hosted engine.  Once nothing else is running (but the host itself), only
> then does the network seem to return to normal.
>
> I'm really at my wits end on this; I have no idea what is causing this
> or what might have changed to cause the issue right at that time.  I
> also can't imagine what ovirt is doing over the network that could cause
> the modem, two physical hops away, to lose its mind in this way.  But my
> experiementation is definitely showing a direct correlation.
>
> Help!!
>
> -derek
>
> --
>Derek Atkins 617-623-3745 <(617)%20623-3745>
>de...@ihtfp.com www.ihtfp.com
>Computer and Internet Security Consultant
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Proper Network Configuration

[~Stack~]

Greetings,

For various reasons I have multiple networks that I am required to work
with. I just want to ensure that I've understood the documentation for
setting up Ovirt correctly.

- First is my BMC/ilo network. The security team wants as few entry
points into this as possible and wants as much segregation as possible.

- Second is my "management" access network. For my other machines on
this network this means admin-SSH/rsyslog/SaltStack configuration
management/ect.

- Third is my high speed network where my NFS storage sits and
applications that need the bandwidth do their thing.

- Fourth is my "public" access.

My Engine host has the "management" and "public" networks.
My Hypervisor hosts have the "BMC/ilo", "management", and "storage"
networks.

Is there a reason why I should add "public" on the hypervisors?

Is there a reason why I may need "BMC/ilo" or "storage" on the Engine host?

Thanks!
~Stack~



signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Help with Power Management network

[~Stack~]

On 09/29/2017 05:31 PM, Dan Yasny wrote:
> You need more than one host for power management

Thanks for the help on this. Added a second host and had IMPI working in
minutes.

~Stack~



signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] Ovirt causing strange network issues?

[Derek Atkins]

Hi,

I'm at my wits end so I'm tossing this here in the hopes that SOMEONE
will be able to help me.

tl;dr: Ovirt is doing something on my network that is causing my fiber
modem to go from 3-5ms to 300-1000+ms round trip times.  I know it's
ovirt because when I unplug ovirt from my network the issue goes away;
when I plug it back in, the issue recurs.

Long version:

I've been running Ovirt 4.0.6 happily on CentOS 7.3 for several months
on a single host machine. Indeed, the host had an uptime of 200+ days
and was working great until approximately midnight, September 21/22
(just over a week ago).  I was on an airplane halfway across the
Atlantic at that time, so it wasn't anything I did.

My network is configured as:

  fiber modem <-> edgerouter <-> switch <-> everything else

ovirt is living in the "everything else" area.

When I sit with a laptop connected to either the everything else range
or even directly connected to the fiber modem, I run 'mtr' and see
network times (starting at the fiber modem) that bounce all over the
place.  When I unplug ovirt I see consistent 3-5ms times.  Plug it back
in, voom, back up to badness.

I've spent several hours plugging and unplugging different devices
trying to isolate the issue.  The only "device" that has any effect is
my ovirt box.

I have tried to debug this in several ways, but really the only thing
that seems to have helped at all is shutting down all the VMs and the
hosted engine.  Once nothing else is running (but the host itself), only
then does the network seem to return to normal.

I'm really at my wits end on this; I have no idea what is causing this
or what might have changed to cause the issue right at that time.  I
also can't imagine what ovirt is doing over the network that could cause
the modem, two physical hops away, to lose its mind in this way.  But my
experiementation is definitely showing a direct correlation.

Help!!

-derek

-- 
   Derek Atkins 617-623-3745
   de...@ihtfp.com www.ihtfp.com
   Computer and Internet Security Consultant
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine setup question

[Charles Kozler]

I did a 3.6 to 4.1 like this. I moved all of my VMs to a new storage domain
(the other was hyperconverged gluster) and then took a full outage, shut
down all of my VMs, detached from 3.6, and imported on 4.1. I had no issues
other than expected mac address changes, but I think you can manually
override this in the engine somewhere

If you are worried, do it with one VM. Create a new storage domain that
both clusters can "see", move one VM to the domain on 3.6, detach, and
import to 3.1. Bring the VM up

If it is Linux VM's older than systemd and using sysvinit, you will hit
issues where your MAC address will change and udev will move it to eth#
wherever # is the next available NIC in your VM host

On Mon, Oct 2, 2017 at 12:54 PM, Demeter Tibor  wrote:

> Hi,
> Can anyone answer my questions?
>
> Thanks in advance,
> R,
>
> Tibor
>
> - 2017. szept.. 19., 8:31, Demeter Tibor  írta:
>
>
> - I have a productive ovirt cluster based on 3.5 series. This using a
> shared nfs storage.  Is it possible to migrate VMs from 3.5 to 4.1 with
> detach shared storage from the old cluster and attach it to the new
> cluster?
> - If yes what will happend with the VM properies? For example mac
> addresses, limits, etc. Those will be migrated or not?
>
> Thanks in advance,
> Regard
>
>
> Tibor
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Hosted engine setup question

[Demeter Tibor]

Hi, 
Can anyone answer my questions? 

Thanks in advance, 
R, 

Tibor 

- 2017. szept.. 19., 8:31, Demeter Tibor  írta: 

> - I have a productive ovirt cluster based on 3.5 series. This using a shared 
> nfs
> storage. Is it possible to migrate VMs from 3.5 to 4.1 with detach shared
> storage from the old cluster and attach it to the new cluster?
> - If yes what will happend with the VM properies? For example mac addresses,
> limits, etc. Those will be migrated or not?

> Thanks in advance,
> Regard

> Tibor

> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] deprecating export domain?

[shubham dubey]

Yes, I just gave an example case. If you want to use vms with a backup,
then you can just copy that vm or disk into another domain and make it as
backup domain as you do it in export domain.
In simple language, main aim of creating backup domain is just to use all
the features available in export domain without creating a dedicated export
domain.
Hope you understand now:)



On 2 Oct 2017 8:55 pm, "Pavel Gashev"  wrote:

> Shubham,
>
>
>
> I don’t really understand the process you described. If I need to backup
> the whole datacenter, you say I have to turn off all VMs, and make them
> non-runnable. It doesn’t look like a backing up. It looks like an
> archiving. But what if I need to keep my VMs running?
>
>
>
>
>
> *From: *shubham dubey 
> *Date: *Monday, 2 October 2017 at 15:55
> *To: *Charles Kozler 
> *Cc: *Pavel Gashev , users , Maor
> Lipchuk 
> *Subject: *Re: [ovirt-users] deprecating export domain?
>
>
>
> Hi,
>
> The backup storage domain is quite like export storage domain but with
> easy usability.
>
> Since you can change any data domain into backup domain any time, you not
> need to create a dedicated
>
> export storage domain for backup or disaster recovery purpose. Altough its
> working is same as export sd.
>
> The process of backup can be as simple as this:
>
> 1) turn off all the vms in your storage domain
>
> 2) select backup flag to convert that into backup domain.
>
> Once the domain is used for backup, you cannot make any changes to its
> vms, disk etc as mentioned by maor.
>
> And yes, you can convert export sd to data domain using cli script but it
> is not required anymore.
>
> If in future export storage domain get deprecated, you not need to be
> worry about that much since you can convert all your
>
> export sd into data domain anytime and start using backup feature instead.
>
> Regards,
>
> Shubham
>
>
>
>
>
> On Mon, Oct 2, 2017 at 6:04 PM, Charles Kozler 
> wrote:
>
> Thank you for clearing this up for me everyone. My concern that something
> like the export domain wasnt going to exist and it was just going to be
> deprecated with no alternative. Glad to hear all the news of the SD
>
>
>
> On Mon, Oct 2, 2017 at 8:31 AM, Pavel Gashev  wrote:
>
> Maor,
>
> Could you please clarify, what would be the process of making backup of a
> running VM to an existing backup storage domain?
>
> I’m asking because it looks like the process is going to be quite the same:
> 1. Clone VM from a snapshot
> 2. Move the cloned VM to a backup storage domain
>
> An ability of choosing destination storage for cloned VMs would increase
> backup efficiency. On the other hand, an ability of exporting VM from a
> snapshot would increase the efficiency in the same way even without
> creating new entity.
>
> Indeed, Backup SDs would increase efficiency of disaster recovery. But the
> same would be achieved by converting Export SDs to Data SDs using a small
> CLI utility.
>
>
> On 01/10/2017, 15:32, "users-boun...@ovirt.org on behalf of Maor Lipchuk"
>  wrote:
>
> On Sun, Oct 1, 2017 at 2:50 PM, Nir Soffer  wrote:
> >
> > Attaching and detaching data domain was not designed for backing up
> vms.
> > How would you use it for backup?
> >
> > How do you ensure that a backup clone of a vm is not started by
> mistake,
> > changing the backup contents?
>
> That is a good question.
> We recently introduced a new feature called "backup storage domain"
> which you can mark the storage domain as backup storage domain.
> That can guarantee that no VMs will run with disks/leases reside on
> the storage domain.
> The feature should already exist in oVirt 4.2 (despite a bug that
> should be handled with this patch https://gerrit.ovirt.org/#/c/81290/)
> You can find more information on this here:
>   https://github.com/shubham0d/ovirt-site/blob/
> 41dcb0f1791d90d1ae0ac43cd34a399cfedf54d8/source/develop/
> release-management/features/storage/backup-storage-domain.html.md
>
> Basically the OVF that is being saved in the export domain should be
> similar to the same one that is being saved in the OVF_STORE disk in
> the storage domain.
> If the user manages replication on that storage domain it can be
> re-used for backup purposes by importing it to a setup.
> Actually it is much more efficient to use a data storage domain than
> to use the copy operation to/from the export storage domain.
>
> >
> > Nir
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> 

Re: [ovirt-users] xfs fragmentation problem caused data domain to hang

[Jason Keltz]

On 10/02/2017 11:05 AM, Jason Keltz wrote:

On 10/02/2017 11:00 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:57 PM, Jason Keltz > wrote:



On 10/02/2017 10:51 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:14 PM, Jason Keltz > wrote:


On 10/02/2017 01:22 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:11 AM, Jason Keltz
> wrote:

Hi.

For my data domain, I have one NFS server with a large
RAID filesystem (9 TB).
I'm only using 2 TB of that at the moment. Today, my
NFS server  hung with
the following error:

xfs: possible memory allocation deadlock in kmem_alloc


Can you share more of the log so we'll see what happened
before and after?
Y.


Here is engine-log from yesterday.. the problem started
around 14:29 PM.
http://www.eecs.yorku.ca/~jas/ovirt-debug/10012017/engine-log.txt



Here is the vdsm log on one of the virtualization
hosts, virt01:
http://www.eecs.yorku.ca/~jas/ovirt-debug/10012017/vdsm.log.2


Doing further investigation, I found that the XFS error
messages didn't start yesterday.  You'll see they
started at the very end of the day on September 23.  See:

http://www.eecs.yorku.ca/~jas/ovirt-debug/messages-20170924




Our storage guys do NOT think it's an XFS fragmentation
issue, but we'll be looking at it.


This is an interesting thread to read because the problem sounds quite 
similar:


http://oss.sgi.com/archives/xfs/2016-03/msg00447.html

In particular, quoted from that:

XFS maintains the full extent list for an active inode in memory,
As it is, yes, the memory allocation problem is with the in-core
extent tree, and we've known about it for some time. The issue is
that as memory gets fragmented, the top level indirection array
grows too large to be allocated as a contiguous chunk. When this
happens really depends on memory load, uptime and the way the extent
tree is being modified.


So in my case, I have a bunch of big XFS disk images for virtual disks.  
As the files are big with many extents, keeping all that information in 
memory at the same time may be the culprit.   Having many extents per se 
isn't the problem, but having enough memory to be able to store all the 
information simultaneously may be. Possible solutions would be to 
increase the default extent size of the volume (which I'm not sure how 
to do), defragment the disk, and hence less extents, or potentially add 
more memory to the file server.  It has 64G.



Hmmm... almost sorry to hear that because that would be easy
to "fix"...



They continued on the 24th, then on the 26th... I think
there were a few "hangs" on those times that people
were complaining about, but we didn't catch the
problem. However, the errors hit big time yesterday at
14:27 PM... see here:

http://www.eecs.yorku.ca/~jas/ovirt-debug/messages-20171001


If you want any other logs, I'm happy to provide them. 
I just don't know exactly what to provide.


Do you know if I can run the XFS defrag command live?
Rather than on a disk by disk, I'd rather just do it on
the whole filesystem.  There really aren't that many
files since it's just ovirt disk images.  However, I
don't understand the implications to running VMs.  I
wouldn't want to do anything to create more downtime.


Should be enough to copy the disks to make them less
fragmented.

Yes, but this requires downtime.. but there's plenty of
additional storage, so this would fix things well.



Live storage migration could be used.
Y.





I had upgraded the engine server + 4 virtualization hosts
from 4.1.1 to current on September 20 along with upgrading
them from CentOS 7.3 to CentOS 7.4.  virtfs, the NFS file
server, was running CentOS 7.3 and kernel
vmlinuz-3.10.0-514.16.1.el7.x86_64. Only yesterday, did I
upgrade it to CentOS 7.4 and hence kernel
vmlinuz-3.10.0-693.2.2.el7.x86_64.

I believe the problem is fully XFS related, and not ovirt at
all. Although, I must admit, ovirt didn't help either.  When
I rebooted the file server, the iso and export domains were

Re: [ovirt-users] libvirt: XML-RPC error : authentication failed: Failed to start SASL

[VONDRA Alain]

Hello,
Thank you very much Yaniv, I willl try it as soon as possible, I’ve made a 
restoration first, to have back the hypervisor quickly.
Regards






Alain VONDRA
Chargé d'Exploitation et de Sécurité des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr





[cid:signature_Unicef-France_ade377bf-317d-4502-9e2f-a0b487c09563.gif]










De : Yaniv Kaul [mailto:yk...@redhat.com]
Envoyé : jeudi 28 septembre 2017 09:39
À : VONDRA Alain 
Cc : Ozan Uzun ; Ovirt Users 
Objet : Re: [ovirt-users] libvirt: XML-RPC error : authentication failed: 
Failed to start SASL



On Wed, Sep 27, 2017 at 7:01 PM, VONDRA Alain 
> wrote:
Hello,
I have exactly the same problem after an upgrade from CentOS 7.3 to 7.4, but I 
don’t want to plan now the migration to oVirt 4.x.
Can you help me to correct the bug and keep oVirt 3.6 for a few months ?
It really seems to be a modification in libvirt authentication because when I 
comment out
#auth_unix_rw="sasl"
in libvirtd.conf, libvirtd starts but my Host is still unresponsive in oVirt.
My production environment  is running on a single Hypervisor and I need the 
second one.
Thanks

The fix is[1]. I suppose you need to change:
mech_list: scram-sha-1

Y.

[1] https://gerrit.ovirt.org/#/c/76934/


De : users-boun...@ovirt.org 
[mailto:users-boun...@ovirt.org] De la part de 
Yaniv Kaul
Envoyé : mardi 19 septembre 2017 13:36
À : Ozan Uzun >
Cc : Ovirt Users >
Objet : Re: [ovirt-users] libvirt: XML-RPC error : authentication failed: 
Failed to start SASL



On Tue, Sep 19, 2017 at 12:24 PM, Ozan Uzun 
> wrote:



Alain VONDRA
Chargé d'Exploitation et de Sécurité des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay 
Trouin
  75006 
PARIS
www.unicef.fr


[cid:image001.gif@01D33BA4.C26AA280]








After hours of struggle, I removed all the hosts.
Installed a fresh centos 6.x on a host. Now it works like a charm.

I will install a fresh ovirt 4.x, and start migration my vm's on new centos 7.4 
hosts.

The only supported way seems exporting/importing vm's for different ovirt 
engines. I wish  I had plain  qcow2 images to copy...


You could detach and attach a whole storage domain.
Y.


On Tue, 19 Sep 2017 at 10:18, Yaniv Kaul 
> wrote:
On Mon, Sep 18, 2017 at 11:47 PM, Ozan Uzun 
> wrote:
Hello,
Today I updated my ovirt engine v3.5 and all my hosts on one datacenter (centos 
7.4 ones).

You are mixing an ancient release (oVirt 3.5) with the latest CentOS. This is 
not supported at best, and who knows if it works.

and suddenly  my vdsm and vdsm-network  services stopped working.
btw: My other DC is centos 6 based (managed from the same ovirt engine), 
everything works just fine there.

vdsm fails dependent on vdsm-network service, with lots of RPC error.
I tried to configure vdsm-tool configure --force, deleted everything 
(vdsm-libvirt), reinstalled.
Could not make it work.
My logs are filled with the follogin

Sep 18 23:06:01 node6 python[5340]: GSSAPI Error: Unspecified GSS failure.  
Minor code may provide more information (No Kerberos credentials available 
(default cache: KEYRING:persistent:0))

This may sound like a change that happened in libvirt authentication, which 
we've adjusted to in oVirt 4.1.5 (specifically VDSM) I believe.
Y.

Sep 18 23:06:01 node6 vdsm-tool[5340]: libvirt: XML-RPC error : authentication 
failed: Failed to start SASL negotiation: -1 (SASL(-1): generic failure: GSSAPI 
Error: Unspecified GSS failure.  Minor code may provide more information (No 
Kerberos credent
Sep 18 23:06:01 node6 libvirtd[4312]: 2017-09-18 20:06:01.954+: 4312: error 
: virNetSocketReadWire:1808 : End of file while reading data: Input/output error

---
journalctl -xe output for vdsm-network

Sep 18 23:06:02 node6 vdsm-tool[5340]: libvirt: XML-RPC error : authentication 
failed: Failed to start SASL negotiation: -1 (SASL(-1): generic failure: GSSAPI 
Error: Unspecified GSS failure.  Minor code may provide more information 

Re: [ovirt-users] deprecating export domain?

[Pavel Gashev]

Shubham,

I don’t really understand the process you described. If I need to backup the 
whole datacenter, you say I have to turn off all VMs, and make them 
non-runnable. It doesn’t look like a backing up. It looks like an archiving. 
But what if I need to keep my VMs running?


From: shubham dubey 
Date: Monday, 2 October 2017 at 15:55
To: Charles Kozler 
Cc: Pavel Gashev , users , Maor Lipchuk 

Subject: Re: [ovirt-users] deprecating export domain?

Hi,
The backup storage domain is quite like export storage domain but with easy 
usability.
Since you can change any data domain into backup domain any time, you not need 
to create a dedicated
export storage domain for backup or disaster recovery purpose. Altough its 
working is same as export sd.
The process of backup can be as simple as this:
1) turn off all the vms in your storage domain
2) select backup flag to convert that into backup domain.
Once the domain is used for backup, you cannot make any changes to its vms, 
disk etc as mentioned by maor.
And yes, you can convert export sd to data domain using cli script but it is 
not required anymore.
If in future export storage domain get deprecated, you not need to be worry 
about that much since you can convert all your
export sd into data domain anytime and start using backup feature instead.
Regards,
Shubham


On Mon, Oct 2, 2017 at 6:04 PM, Charles Kozler 
> wrote:
Thank you for clearing this up for me everyone. My concern that something like 
the export domain wasnt going to exist and it was just going to be deprecated 
with no alternative. Glad to hear all the news of the SD

On Mon, Oct 2, 2017 at 8:31 AM, Pavel Gashev 
> wrote:
Maor,

Could you please clarify, what would be the process of making backup of a 
running VM to an existing backup storage domain?

I’m asking because it looks like the process is going to be quite the same:
1. Clone VM from a snapshot
2. Move the cloned VM to a backup storage domain

An ability of choosing destination storage for cloned VMs would increase backup 
efficiency. On the other hand, an ability of exporting VM from a snapshot would 
increase the efficiency in the same way even without creating new entity.

Indeed, Backup SDs would increase efficiency of disaster recovery. But the same 
would be achieved by converting Export SDs to Data SDs using a small CLI 
utility.


On 01/10/2017, 15:32, "users-boun...@ovirt.org 
on behalf of Maor Lipchuk" 
 on behalf of 
mlipc...@redhat.com> wrote:

On Sun, Oct 1, 2017 at 2:50 PM, Nir Soffer 
> wrote:
>
> Attaching and detaching data domain was not designed for backing up vms.
> How would you use it for backup?
>
> How do you ensure that a backup clone of a vm is not started by mistake,
> changing the backup contents?

That is a good question.
We recently introduced a new feature called "backup storage domain"
which you can mark the storage domain as backup storage domain.
That can guarantee that no VMs will run with disks/leases reside on
the storage domain.
The feature should already exist in oVirt 4.2 (despite a bug that
should be handled with this patch https://gerrit.ovirt.org/#/c/81290/)
You can find more information on this here:
  
https://github.com/shubham0d/ovirt-site/blob/41dcb0f1791d90d1ae0ac43cd34a399cfedf54d8/source/develop/release-management/features/storage/backup-storage-domain.html.md

Basically the OVF that is being saved in the export domain should be
similar to the same one that is being saved in the OVF_STORE disk in
the storage domain.
If the user manages replication on that storage domain it can be
re-used for backup purposes by importing it to a setup.
Actually it is much more efficient to use a data storage domain than
to use the copy operation to/from the export storage domain.

>
> Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] xfs fragmentation problem caused data domain to hang

[Jason Keltz]

On 10/02/2017 11:00 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:57 PM, Jason Keltz > wrote:



On 10/02/2017 10:51 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:14 PM, Jason Keltz > wrote:


On 10/02/2017 01:22 AM, Yaniv Kaul wrote:



On Mon, Oct 2, 2017 at 5:11 AM, Jason Keltz
> wrote:

Hi.

For my data domain, I have one NFS server with a large
RAID filesystem (9 TB).
I'm only using 2 TB of that at the moment. Today, my NFS
server  hung with
the following error:

xfs: possible memory allocation deadlock in kmem_alloc


Can you share more of the log so we'll see what happened
before and after?
Y.


Here is engine-log from yesterday.. the problem started
around 14:29 PM.
http://www.eecs.yorku.ca/~jas/ovirt-debug/10012017/engine-log.txt



Here is the vdsm log on one of the virtualization hosts,
virt01:
http://www.eecs.yorku.ca/~jas/ovirt-debug/10012017/vdsm.log.2


Doing further investigation, I found that the XFS error
messages didn't start yesterday.  You'll see they
started at the very end of the day on September 23. See:

http://www.eecs.yorku.ca/~jas/ovirt-debug/messages-20170924




Our storage guys do NOT think it's an XFS fragmentation
issue, but we'll be looking at it.

Hmmm... almost sorry to hear that because that would be easy
to "fix"...



They continued on the 24th, then on the 26th... I think
there were a few "hangs" on those times that people were
complaining about, but we didn't catch the problem.
However, the errors hit big time yesterday at 14:27
PM... see here:

http://www.eecs.yorku.ca/~jas/ovirt-debug/messages-20171001


If you want any other logs, I'm happy to provide them. I
just don't know exactly what to provide.

Do you know if I can run the XFS defrag command live?
Rather than on a disk by disk, I'd rather just do it on
the whole filesystem. There really aren't that many
files since it's just ovirt disk images.  However, I
don't understand the implications to running VMs.  I
wouldn't want to do anything to create more downtime.


Should be enough to copy the disks to make them less fragmented.

Yes, but this requires downtime.. but there's plenty of
additional storage, so this would fix things well.



Live storage migration could be used.
Y.





I had upgraded the engine server + 4 virtualization hosts
from 4.1.1 to current on September 20 along with upgrading
them from CentOS 7.3 to CentOS 7.4.  virtfs, the NFS file
server, was running CentOS 7.3 and kernel
vmlinuz-3.10.0-514.16.1.el7.x86_64. Only yesterday, did I
upgrade it to CentOS 7.4 and hence kernel
vmlinuz-3.10.0-693.2.2.el7.x86_64.

I believe the problem is fully XFS related, and not ovirt at
all.   Although, I must admit, ovirt didn't help either. When
I rebooted the file server, the iso and export domains were
immediately active, but the data domain took quite a long
time.  I kept trying to activate it, and it couldn't do it. 
I couldn't make a host an SPM.  I found that the data domain

directory on the virtualization host was a "stale NFS file
handle".  I rebooted one of the virtualization hosts (virt1),
and tried to make it the SPM.  Again, it wouldn't work. 
Finally, I ended up turning everything into maintenance mode,
then activating just it, and I was able to make it the SPM. 
I was then able to bring everything up.  I would have

expected ovirt to handle the problem a little more
gracefully, and give me more information because I was
sweating thinking I had to restore all the VMs!


Stale NFS is on our todo list to handle. Quite challenging.

Thanks..



I didn't think when I chose XFS as the filesystem for my
virtualization NFS server that I would have to defragment the
filesystem manually.  This is like the old days of running
Norton SpeedDisk to defrag my 386...


We are still not convinced it's an issue - but we'll look into it
(and perhaps ask for more 

[ovirt-users] SPM recovery after disaster

[Alexander Vrublevskiy]

Hello Community!
Recently we had a disaster with our oVirt 4.1 three nodes cluster 
with HE and GlusterFS (RF=3) storage domain. We've moved one node to 
maintenance and during actual maintenance one of working nodes with SPM 
role went down. It was hardware failure so we had to remove it from the 
cluster.
After tinkering around now we have almost working cluster with two 
nodes and with GlusterFS RF=2. But the problem is oVirt can't find SPM 
and spaming web interface logs with "HSMGetAllTasksStatusesVDS failed: 
Not SPM" error.
After some time of operating with stated configuration we lost contents of 
dom_md somehow.
Looks like these two problems are related and second one is a consequence of 
the first.
Please suggest how to recover SPM and dom_md. Is there a way to recreate both?
TIA
Regards
Alex___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Having issue with external IPA

[Donny Davis]

Does the user you are using have permissions to add people? Why not just
use IPA to add the user accounts?

On Mon, Oct 2, 2017 at 2:26 AM, Ondra Machacek  wrote:

> On Sun, Oct 1, 2017 at 1:07 PM, Yan Naing Myint
>  wrote:
> > Hello guys,
> >
> > I'm having problem with adding users from my FreeIPA server to oVirt.
> > 1. Status of ovirt-engine-extension-aaa-ldap-setup is success with RHDS
> > 2. I cannot add IPA users in oVirt webadmin panel
> > 3. In oVirt web admin panel it says "Error while executing action
> AddUser:
> > Internal Engine Error"
> >
> > What will be the problem or is it a bug?
>
> Can you please share the log from the following command?
>
>  $ ovirt-engine-extensions-tool --log-level=FINEST
> --log-file=/tmp/aaa.log aaa search --entity-name=mgorca
> --extension-name=cyberwings.local
>
> > Is there any suggestion of how do it make it work?
> >
> > in the engine.log it says;
> >
> > 2017-10-01 17:30:52,436+06 ERROR
> > [org.ovirt.engine.core.bll.aaa.AddUserCommand] (default task-113)
> > [bf5822eb-39da-49e5-b2ab-9865f71346a3] Transaction rolled-back for
> command
> > 'org.ovirt.engine.core.bll.aaa.AddUserCommand'.
> > 2017-10-01 17:30:52,459+06 WARN
> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> > (default task-113) [bf5822eb-39da-49e5-b2ab-9865f71346a3] EVENT_ID:
> > USER_FAILED_ADD_ADUSER(327), Correlation ID:
> > bf5822eb-39da-49e5-b2ab-9865f71346a3, Call Stack: null, Custom ID: null,
> > Custom Event ID: -1, Message: Failed to add User 'mgorca' to the system.
> >
> > in cyberwings.local.properties
> >
> > ovirt.engine.extension.name = cyberwings.local
> > ovirt.engine.extension.bindings.method = jbossmodule
> > ovirt.engine.extension.binding.jbossmodule.module =
> > org.ovirt.engine-extensions.aaa.ldap
> > ovirt.engine.extension.binding.jbossmodule.class =
> > org.ovirt.engineextensions.aaa.ldap.AuthzExtension
> > ovirt.engine.extension.provides = org.ovirt.engine.api.
> extensions.aaa.Authz
> > config.profile.file.1 = ../aaa/cyberwings.local.properties
> > config.globals.baseDN.simple_baseDN = dc=cyberwings,dc=local
> >
> > in cyberwings.local-authn.properties
> > ovirt.engine.extension.name = cyberwings.local-authn
> > ovirt.engine.extension.bindings.method = jbossmodule
> > ovirt.engine.extension.binding.jbossmodule.module =
> > org.ovirt.engine-extensions.aaa.ldap
> > ovirt.engine.extension.binding.jbossmodule.class =
> > org.ovirt.engineextensions.aaa.ldap.AuthnExtension
> > ovirt.engine.extension.provides = org.ovirt.engine.api.
> extensions.aaa.Authn
> > ovirt.engine.aaa.authn.profile.name = cyberwings.local
> > ovirt.engine.aaa.authn.authz.plugin = cyberwings.local
> > config.profile.file.1 = ../aaa/cyberwings.local.properties
> > config.globals.baseDN.simple_baseDN = dc=cyberwings,dc=local
> >
> >
> > --
> > Yan Naing Myint
> > CEO
> > Server & Network Engineer
> > Cyber Wings Co., Ltd
> > http://cyberwings.asia
> > 09799950510
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] deprecating export domain?

[shubham dubey]

Hi,
The backup storage domain is quite like export storage domain but with easy
usability.
Since you can change any data domain into backup domain any time, you not
need to create a dedicated
export storage domain for backup or disaster recovery purpose. Altough its
working is same as export sd.

The process of backup can be as simple as this:
1) turn off all the vms in your storage domain
2) select backup flag to convert that into backup domain.

Once the domain is used for backup, you cannot make any changes to its vms,
disk etc as mentioned by maor.
And yes, you can convert export sd to data domain using cli script but it
is not required anymore.

If in future export storage domain get deprecated, you not need to be worry
about that much since you can convert all your
export sd into data domain anytime and start using backup feature instead.

Regards,
Shubham


On Mon, Oct 2, 2017 at 6:04 PM, Charles Kozler  wrote:

> Thank you for clearing this up for me everyone. My concern that something
> like the export domain wasnt going to exist and it was just going to be
> deprecated with no alternative. Glad to hear all the news of the SD
>
> On Mon, Oct 2, 2017 at 8:31 AM, Pavel Gashev  wrote:
>
>> Maor,
>>
>> Could you please clarify, what would be the process of making backup of a
>> running VM to an existing backup storage domain?
>>
>> I’m asking because it looks like the process is going to be quite the
>> same:
>> 1. Clone VM from a snapshot
>> 2. Move the cloned VM to a backup storage domain
>>
>> An ability of choosing destination storage for cloned VMs would increase
>> backup efficiency. On the other hand, an ability of exporting VM from a
>> snapshot would increase the efficiency in the same way even without
>> creating new entity.
>>
>> Indeed, Backup SDs would increase efficiency of disaster recovery. But
>> the same would be achieved by converting Export SDs to Data SDs using a
>> small CLI utility.
>>
>>
>> On 01/10/2017, 15:32, "users-boun...@ovirt.org on behalf of Maor
>> Lipchuk" 
>> wrote:
>>
>> On Sun, Oct 1, 2017 at 2:50 PM, Nir Soffer 
>> wrote:
>> >
>> > Attaching and detaching data domain was not designed for backing up
>> vms.
>> > How would you use it for backup?
>> >
>> > How do you ensure that a backup clone of a vm is not started by
>> mistake,
>> > changing the backup contents?
>>
>> That is a good question.
>> We recently introduced a new feature called "backup storage domain"
>> which you can mark the storage domain as backup storage domain.
>> That can guarantee that no VMs will run with disks/leases reside on
>> the storage domain.
>> The feature should already exist in oVirt 4.2 (despite a bug that
>> should be handled with this patch https://gerrit.ovirt.org/#/c/81290/
>> )
>> You can find more information on this here:
>>   https://github.com/shubham0d/ovirt-site/blob/41dcb0f1791d90d
>> 1ae0ac43cd34a399cfedf54d8/source/develop/release-
>> management/features/storage/backup-storage-domain.html.md
>>
>> Basically the OVF that is being saved in the export domain should be
>> similar to the same one that is being saved in the OVF_STORE disk in
>> the storage domain.
>> If the user manages replication on that storage domain it can be
>> re-used for backup purposes by importing it to a setup.
>> Actually it is much more efficient to use a data storage domain than
>> to use the copy operation to/from the export storage domain.
>>
>> >
>> > Nir
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] deprecating export domain?

[Charles Kozler]

Thank you for clearing this up for me everyone. My concern that something
like the export domain wasnt going to exist and it was just going to be
deprecated with no alternative. Glad to hear all the news of the SD

On Mon, Oct 2, 2017 at 8:31 AM, Pavel Gashev  wrote:

> Maor,
>
> Could you please clarify, what would be the process of making backup of a
> running VM to an existing backup storage domain?
>
> I’m asking because it looks like the process is going to be quite the same:
> 1. Clone VM from a snapshot
> 2. Move the cloned VM to a backup storage domain
>
> An ability of choosing destination storage for cloned VMs would increase
> backup efficiency. On the other hand, an ability of exporting VM from a
> snapshot would increase the efficiency in the same way even without
> creating new entity.
>
> Indeed, Backup SDs would increase efficiency of disaster recovery. But the
> same would be achieved by converting Export SDs to Data SDs using a small
> CLI utility.
>
>
> On 01/10/2017, 15:32, "users-boun...@ovirt.org on behalf of Maor Lipchuk"
>  wrote:
>
> On Sun, Oct 1, 2017 at 2:50 PM, Nir Soffer  wrote:
> >
> > Attaching and detaching data domain was not designed for backing up
> vms.
> > How would you use it for backup?
> >
> > How do you ensure that a backup clone of a vm is not started by
> mistake,
> > changing the backup contents?
>
> That is a good question.
> We recently introduced a new feature called "backup storage domain"
> which you can mark the storage domain as backup storage domain.
> That can guarantee that no VMs will run with disks/leases reside on
> the storage domain.
> The feature should already exist in oVirt 4.2 (despite a bug that
> should be handled with this patch https://gerrit.ovirt.org/#/c/81290/)
> You can find more information on this here:
>   https://github.com/shubham0d/ovirt-site/blob/
> 41dcb0f1791d90d1ae0ac43cd34a399cfedf54d8/source/develop/
> release-management/features/storage/backup-storage-domain.html.md
>
> Basically the OVF that is being saved in the export domain should be
> similar to the same one that is being saved in the OVF_STORE disk in
> the storage domain.
> If the user manages replication on that storage domain it can be
> re-used for backup purposes by importing it to a setup.
> Actually it is much more efficient to use a data storage domain than
> to use the copy operation to/from the export storage domain.
>
> >
> > Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] deprecating export domain?

[Pavel Gashev]

Maor,

Could you please clarify, what would be the process of making backup of a 
running VM to an existing backup storage domain?

I’m asking because it looks like the process is going to be quite the same:
1. Clone VM from a snapshot
2. Move the cloned VM to a backup storage domain

An ability of choosing destination storage for cloned VMs would increase backup 
efficiency. On the other hand, an ability of exporting VM from a snapshot would 
increase the efficiency in the same way even without creating new entity. 

Indeed, Backup SDs would increase efficiency of disaster recovery. But the same 
would be achieved by converting Export SDs to Data SDs using a small CLI 
utility.


On 01/10/2017, 15:32, "users-boun...@ovirt.org on behalf of Maor Lipchuk" 
 wrote:

On Sun, Oct 1, 2017 at 2:50 PM, Nir Soffer  wrote:
>
> Attaching and detaching data domain was not designed for backing up vms.
> How would you use it for backup?
>
> How do you ensure that a backup clone of a vm is not started by mistake,
> changing the backup contents?

That is a good question.
We recently introduced a new feature called "backup storage domain"
which you can mark the storage domain as backup storage domain.
That can guarantee that no VMs will run with disks/leases reside on
the storage domain.
The feature should already exist in oVirt 4.2 (despite a bug that
should be handled with this patch https://gerrit.ovirt.org/#/c/81290/)
You can find more information on this here:
  
https://github.com/shubham0d/ovirt-site/blob/41dcb0f1791d90d1ae0ac43cd34a399cfedf54d8/source/develop/release-management/features/storage/backup-storage-domain.html.md

Basically the OVF that is being saved in the export domain should be
similar to the same one that is being saved in the OVF_STORE disk in
the storage domain.
If the user manages replication on that storage domain it can be
re-used for backup purposes by importing it to a setup.
Actually it is much more efficient to use a data storage domain than
to use the copy operation to/from the export storage domain.

>
> Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Having issue with external IPA

[Ondra Machacek]

On Sun, Oct 1, 2017 at 1:07 PM, Yan Naing Myint
 wrote:
> Hello guys,
>
> I'm having problem with adding users from my FreeIPA server to oVirt.
> 1. Status of ovirt-engine-extension-aaa-ldap-setup is success with RHDS
> 2. I cannot add IPA users in oVirt webadmin panel
> 3. In oVirt web admin panel it says "Error while executing action AddUser:
> Internal Engine Error"
>
> What will be the problem or is it a bug?

Can you please share the log from the following command?

 $ ovirt-engine-extensions-tool --log-level=FINEST
--log-file=/tmp/aaa.log aaa search --entity-name=mgorca
--extension-name=cyberwings.local

> Is there any suggestion of how do it make it work?
>
> in the engine.log it says;
>
> 2017-10-01 17:30:52,436+06 ERROR
> [org.ovirt.engine.core.bll.aaa.AddUserCommand] (default task-113)
> [bf5822eb-39da-49e5-b2ab-9865f71346a3] Transaction rolled-back for command
> 'org.ovirt.engine.core.bll.aaa.AddUserCommand'.
> 2017-10-01 17:30:52,459+06 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-113) [bf5822eb-39da-49e5-b2ab-9865f71346a3] EVENT_ID:
> USER_FAILED_ADD_ADUSER(327), Correlation ID:
> bf5822eb-39da-49e5-b2ab-9865f71346a3, Call Stack: null, Custom ID: null,
> Custom Event ID: -1, Message: Failed to add User 'mgorca' to the system.
>
> in cyberwings.local.properties
>
> ovirt.engine.extension.name = cyberwings.local
> ovirt.engine.extension.bindings.method = jbossmodule
> ovirt.engine.extension.binding.jbossmodule.module =
> org.ovirt.engine-extensions.aaa.ldap
> ovirt.engine.extension.binding.jbossmodule.class =
> org.ovirt.engineextensions.aaa.ldap.AuthzExtension
> ovirt.engine.extension.provides = org.ovirt.engine.api.extensions.aaa.Authz
> config.profile.file.1 = ../aaa/cyberwings.local.properties
> config.globals.baseDN.simple_baseDN = dc=cyberwings,dc=local
>
> in cyberwings.local-authn.properties
> ovirt.engine.extension.name = cyberwings.local-authn
> ovirt.engine.extension.bindings.method = jbossmodule
> ovirt.engine.extension.binding.jbossmodule.module =
> org.ovirt.engine-extensions.aaa.ldap
> ovirt.engine.extension.binding.jbossmodule.class =
> org.ovirt.engineextensions.aaa.ldap.AuthnExtension
> ovirt.engine.extension.provides = org.ovirt.engine.api.extensions.aaa.Authn
> ovirt.engine.aaa.authn.profile.name = cyberwings.local
> ovirt.engine.aaa.authn.authz.plugin = cyberwings.local
> config.profile.file.1 = ../aaa/cyberwings.local.properties
> config.globals.baseDN.simple_baseDN = dc=cyberwings,dc=local
>
>
> --
> Yan Naing Myint
> CEO
> Server & Network Engineer
> Cyber Wings Co., Ltd
> http://cyberwings.asia
> 09799950510
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users