[ovirt-users] Re: Moving from CentOS8 to CentOS9 based OvirtNodes NG

[Thomas Hoberg]

Simon Coter just told me, I'm all wrong and that 4.5 still supports HCI as well 
as both kernels.

So, please test and prove me wrong!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LXQNS43XKSQHC6UUX5BIV5HN7Q7N3C65/

[ovirt-users] Re: [External] : Re: Ovirt Hyperconverged

[Thomas Hoberg]

Hi Simon!

I'd given up on ever finding any real person or back-channel on the Oracle side 
of oVirt, so you're saying there is actually such a thing!

I'd [have] been more than happy to feed back all those results I was collecting 
in my desperate attempts to maintain a HCI infra with all those shifting 
Enterprise Linux players doing politics.

Today my enterprise use case is transitioning to the cloud and my private use 
case to Proxmox.

The latter has run mostly on Pentium Silver J5005 Atoms during the last couple 
of years and I am currently trying to work out the kinks on KVM live-migration 
between an Orange PI 5+ (32GB) and a Raspberry PI 5 (8GB) under Proxmox (using 
storage on a Ceph HCI cluster running on x86), so you may appreciate why an 
Oracle support contract wasn't in the picture for an infra I keep under four 
digits total invests to appease the wife.

(Ok, I justed noticed that you're running OL9 on your OP5+, but I don't see you 
trying to port oVirt there...)

Without that contract, it seems, that Oracle keeps very "stumm".

So on HCI:

When I ran across oVirt, that was somewhere when 4.3 was fresh and oVirt was 
advertised as "a solution for your entire enterprise" which included HCI, to 
catch potential Nutanix and vSphere customers.

It sold me on the idea, that I could take an oVirt node ISO, install that on my 
hardware nodes, run a GUI wizard to thurn them into a clustered HCI appliance 
and be as happy as the other guy with Nut[ell]anix.

That dream certainly cost me months of my life, not the hours I had imagined, 
but it paid a salary, too, when I managed to run it anyway.

It took me long to realize that Oracle had ditched all their Xen stuff and 
become an oVirt convert. But even since then, there has been very little 
details and firm commitments nor even a branding that doesn't require typing 
classes to execute, so sorry, if most of my impressions are simply from 
informational gaps.

But to my knowledge, Oracle never published node ISO images.

Also to my knowledge, oVirt itself ditched HCI support, Redhat itself made 
nearly the entire technology stack oVirt is based on EOL, Gluster, oVirt, VDO 
and, of course, I had used all of that.

Except storage tiering, where you'd use SSDs in your VDO/Gluster storage for a 
caching layer on top of HDDs: that I never got to work and then SSDs became 
mainline anyway.

On my first EL8/oVirt 4.4 tests Oracle's Enterprise kernel failed immediately 
with VDO, which was missing then. Later even the Redhat kernel sometimes failed 
with VDO after kernel upgrades, because evidently nobody at Oracle cares about 
VDO. Funny, when you consider those Sun guys used to be very big into something 
similar...

But also later I got into all kinds of trouble when I was setting up HCI with 
4.4 and had not switched the kernels to the Redhat variant. If I remember 
correctly, the management engine never managed to connect to the network after 
it had been teleported into KVM and after it had been successfully configured 
locally on the temporary install node. I could have been that I tried this on 
nested virtualization, but it felt more kernel related, because switching that 
fixed the issue.

Later I experimented with upgrades from 4.4 to 4.5 and ran into all sorts of 
trouble when switching the kernel there. Except that now things started failing 
with the Redhat kernel. Generally nobody seems to test switching between UEK 
and RHK on HCI nodes, which *should* be totally transparent on the wire and 
basically to all user space, if I understood Wim Coekaerts correctly, when we 
met in 2011.

So my impression was that Oracle supported a subset of what oVirt supported and 
with HCI not even giving any search responses anywhere on oracle.com, I didn't 
see that remaining.

And perhaps all I missed was to install 'cockpit-ovirt-dashboard'...

So, good to know Oracle hasn't given up, good to know you keep an ear open here 
and now if there was a bit more public commitment for oVirt, we'd all be much 
happier.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KZ6ANPXHSFI2DQYIQP65XOIVA5ENIJJS/

[ovirt-users] Re: Moving from CentOS8 to CentOS9 based OvirtNodes NG

[Thomas Hoberg]

HCI has been deprecated years ago, but somehow the code survived until oVirt 
4.5.5 or so.

Which means it's still present in Oracle's 4.4 derivative. but not in their 4.5 
release.

On that base (make sure to use to switch to the Redhat kernel on all hosts and 
the management engine to avoid troubles) you can at least still build a HCI 
implementation that is in support (EOL not announced by Oracle) and much less 
beta than oVirt releases.

When it comes to do an in-place migration of a HCI setup, open heart surgery 
may have more predictable results. Essentially you'd have to treat it like a 
disaster recovery and follow the path of restoring a management engine from 
backup.

I've gone the migration way, created a new cluster and used a backup domain to 
transfer all VMs.

Gluster in itself is quite independent, which might be a bonus here, while I 
mostly suffered from how it was never really integrated and had many mismatches 
in design philosophy.

It's the storage domains on top that are at risk, but those again are 
relatively independent of the storage underneath, since they were designed for 
SAN, NFS and whatnot. Again, in theory, they can be imported elsewhere, because 
they have metadata describing them.

I've done all my testing using play HCI clusters on VMware Workstation using 
nested virtualization, before I tried things on "live patients".

Whatever you're planning, HCI has basically expired with oVirt 4.4 on EL8. If 
you're happy to loose everything, you may keep trying. I went with Proxmox 
using HCI on Ceph and while it's much more manual in many ways, it has a 
future, is way faster (no Ansible) and rock solid.

Getting VMs migrated is a lot of bother, because none of these orchestrators 
were ever keen on making it easy for customers to defect. But since it's all 
KVM/QEMU underneath, tools that operate at that level will work.

Now you'll just have to learn them or rebuild those VMs: that's "the cloud 
path" anyway...
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HBJR6BM5H3U5NYXW7S467QZLL7LQ2EFA/

[ovirt-users] Re: Ovirt Hyperconverged

[Thomas Hoberg]

I've tried to re-deploy oVirt 4.3 on CentOS7 servers because I had managed to 
utterly destroy a HCI farm, where most VMs had migrated to Oracles variant of 
RHV 4.4 on Oracle Linux. I guess I grew a bit careless towards its end.

Mostly it was just an academic exercise to see if it could be resurrected... I 
was much happier with the Oracle variant anyway.

And I've hit similar issues all over the place: the ansible scripts and/or the 
python packages they interact with are utterly broken with now years of 
completely disjunct bug-fixing going on.

The underlying CentOS 7 packages continue in maintenance (some more weeks to 
go..), but the oVirt 4.3 on top has been unmaintained for years.

Since these are just sanity checks, I deleted all of them, one after the other 
(and there is lots of them!), and I eventually got it to work again.

Don't have a single VM on it, though, because you can't trust it, the hardware 
is ancient and it really was just a finger exercise at that stage. With CentOS 
7 going out of support now, it's really messing with a corpse.

I'm currently operating Oracle's 4.4 variant running on their Linux, too, which 
still has Gluster based HCI built-in, even if they don't mention it at all.

Just make sure you switch their Unbreakable Linux kernel for the Redhat variant 
everywhere, otherwise you'll risk all kinds of nasties.

It's been way more stable than oVirt 4.3 ever was, but that doesn't mean it's 
"enterprise": that was always one fat big exaggeration, withful thinking, 
whatever.

And don't fall for their 4.5 variant, that came out end of last year: that one 
doesn't support HCI any more and actually seems to fail withOUT their 
Enterprise Linux kernels.

And no, it doesn't run on EL9 either, that might take another year or so, as 
Oracle's oVirt implementation is almost a year behind oVirt at the moment.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/X435DYEU5BMHQ7ZHSTJPYF4ZCXQ2GJXU/

[ovirt-users] onn pv

[Michael Thomas]

I think I may have just messed up my cluster.

I'm running an older 4.4.2.6 cluster on CentOS-8 with 4 nodes and a 
self-hosted engine. I wanted to assemble the spare drives on 3 of the 4 
nodes into a new gluster volume for extra VM storage.


Unfortunately, I did not look closely enough at one of the nodes before 
running sfdisk+parted+pvcreate, and now it looks like I may have broken 
my onn storage.  pvs shows missing uuids:


# pvs
  WARNING: Couldn't find device with uuid 
RgTaWg-fR1T-J3Nv-uh03-ZTi5-jz9X-cjl1lo

.
  WARNING: Couldn't find device with uuid 
0l9CFI-Z7pP-x1P8-AJ78-gRoz-ql0e-2gzXsC

.
  WARNING: Couldn't find device with uuid 
fl73h2-ztyn-y9NY-4TF4-K2Pd-G2Ow-vH46yH

.
  WARNING: VG onn_ovirt1 is missing PV 
RgTaWg-fR1T-J3Nv-uh03-ZTi5-jz9X-cjl1lo (l

ast written to /dev/nvme0n1p3).
  WARNING: VG onn_ovirt1 is missing PV 
0l9CFI-Z7pP-x1P8-AJ78-gRoz-ql0e-2gzXsC (l

ast written to /dev/nvme1n1p1).
  WARNING: VG onn_ovirt1 is missing PV 
fl73h2-ztyn-y9NY-4TF4-K2Pd-G2Ow-vH46yH (l

ast written to /dev/nvme2n1p1).
  PV VG   Fmt  Attr PSizePFree
  /dev/md2   vg00 lvm2 a--  <928.80g  0
  /dev/nvme2n1p1 gluster_vg_nvme2n1p1 lvm2 a-- 2.91t  0
  /dev/nvme3n1p1 onn_ovirt1   lvm2 a-- 2.91t  0
  [unknown]  onn_ovirt1   lvm2 a-m   929.92g 100.00g
  [unknown]  onn_ovirt1   lvm2 a-m  <931.51g  0
  [unknown]  onn_ovirt1   lvm2 a-m 2.91t  0


Here's what I don't understand:

* This onn volume group only existed on one of the 4 nodes.  I expected 
it would have been on all 4?


* lsblk and /etc/fstab don't show any reference to onn

* What is the ONN volume group used for, and how bad is it if it's now 
missing?  I note that my VMs all continue to run and I've been able to 
migrate them off of this affected node with no apparent problems.


* Is it possible that this onn volume group was already broken before I 
messed with the nvme3n1 disk?  When ovirt was originally installed 
several years ago, I went through the install process multiple times and 
might not have cleaned up properly each time.


--Mike
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIF3QB2ECLCEDAYICNNZM6Q2S3DB7ROB/

[ovirt-users] Re: Oracle Virtualization Manager 4.5 anyone?

[Thomas Hoberg]

In theory, if oVirt supports it, the Oracle variant would do it to... unless 
they manage to break it.

And since there is zero information on what they test, that could happen at any 
time.

Same for HCI with GlusterFS or VDO. HCI has been removed as "a tested feature", 
but if you use the Cockpit GUI, it will just continue to work quite happily. I 
haven't actually tried it in combination with VDO, though.

I've used oVirt deployed Gluster storage in Proxmox w/o any issue and I'm 
tempted to do the opposite, too (use Proxmox provided Ceph storage in oVirt).

oVirt is all abstractions, so it *should* work. But until somebody actually 
tests, validates and fixes it (if necessary), it's simply a house of cards that 
can break with every change.

Ich würde meine Hoffnungen nicht allzu hoch schrauben, momentan sehe ich das 
Projekt quasi dank seines Trägheitsmoments weiter laufen, aber ohne daß jemand 
das so richtig treibt oder gar Budget bereithält... was ausgesprochen schade 
ist, denn da steck eine riesige Menge Arbeit und weit mehr Potential (für die 
Anwender, weniger für die Hersteller) drinn.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZPGD3UNQPC2WARWRZQLW4ETZADKKUAUE/

[ovirt-users] Re: Oracle Virtualization Manager 4.5 anyone?

[Thomas Hoberg]

> Thomas, your e-mail created too much food for thought... as usual I would
> say, remembering the past ;-)
> I try to reply to some of them online below, putting my own personal
> considerations on the table
> 
Hi Gianluca, nice to meet you again!
> On Thu, Dec 21, 2023 at 11:44 AM Thomas Hoberg  
> 
> I don't think so. I tried for quite some time their previous solution,
> based on Xen, and simply it didn't work as expected in terms of many
> enterprise class needed functionalities. Then I switched to oVirt and/or
> RHV, depending on customer needs. And I think Oracle did the same. If I
> remember correctly there was also a migration path.
Xen is in dire straits. 

Which is a shame, given its history and its sometimes unique qualities e.g in 
the area of Library operating systems...

The biggest technical issue I see is x86 architecture deficits, but in the 
mean-time it's simply eco-system size and some choices like OCAML that have 
turned into one big technical debt that nobody is going to pay.

I think the Xcp-ng guys are heros, on older (supported) hardware Xcp-ng runs 
like a charm, but there are on an extiction branch of virtualization evolution.
> 
> 
> 
> I don't agree. Even if not crystal clear and somehow confusing for
> newcomers, they called it Oracle Linux Virtualization Manager (OLVM) since
> the beginning, making it clear that in their regard they see it as an
> extension of the operating system (Oracle Linux). In fact if you buy the
> Premier Support level of the OS, you automatically get also support for
> OLVM. if you use it.
> Their previous solution branding was Oracle VM (or sometimes Oracle VM
> Server)
> Here you can still find all the solutions described, including VirtualBox
> (and recently Kata Containers):
> https://www.oracle.com/virtualization/
> 
Not a fight I want to pick, but Google fails me.. and I don't think it's 
Google's fault
> 
> 
> In the past I asked and they told me that they planned to continue with
> OLVM and its support even after RHV EOL.
> And the 12th December announcement seems to confirm that.
> Their product is a fork of oVirt.
Without a roadmap and committed life cycles, nothing is a product.
And with oVirt lacking both, Oracle's "fork relation" is nothing to build on.
> 
> 
> 
> Great news, in my opinion
> I didn't spot it yet, but you are right and here you can find the announce
> page:
> https://blogs.oracle.com/virtualization/post/oracle-linux-virtualization-...
> 
It breathes life into what is potentially a carcass with lots of potential... 
But for how long?
> 
> 
> The whole point here is that in my opinion GlusterFS is totally not ready
> for enterprise use, based on my past tests in oVirt context.
> In other terms, possibly the problems you are describing was more dependant
> on GlusterFS configuration/tuning/bugs than on oVirt/OLVM versions
> themselves
> Just my opinion
It was Redhat who spread the fiction of GlusterFS as the one solution without 
any scaling limits.
Your opinion seems to resonate even with Redhat, who decided to discontinue any 
commercial product based on GlusterFS.

But I need HCI, and Redhat/Oracle/oVirt isn't providing an [integrated] 
alternative.
Ceph seems ok for that, but nobody is working on instrumentation.
> 
> 
> 
> I don't think that. Based on licensing they have to let their sources
> publicly available.
> And in fact here you find all you need in case you want to crosscheck:
> https://yum.oracle.com/oracle-linux-8.html
> 
> In the link above you can find both 4.4 and 4.5 sources, together with the
> OS and UEK kernels ones, because as said above, they consider OLVM an
> extension of the operating system functionalities
I've round Oracle Linux sources on Githubu, but their fork of oVirt is nowhere 
to be found...
> 
> 
> 
> They support both Red Hat Compatible kernels and UEK ones. In case of
> problems I think you can submit bugs.
> The correct entry point for that for not paying customers should be this
> one if I'm not wrong:
> https://github.com/oracle/oracle-linux/issues
I've seen that, and it looks like it's being monitored. But the equivalent for 
oVirt or OV is missing!
> 
> 
> 
> I think you well understand that sw developer processing is not an easy
> one, and that it comprises feature freezes and such...
> If Oracle well before planned to come out with a 4.5 release based on a
> fork of a well established 4.5.4 release, it is not so important to rebase
> all the work on the just released oVirt 4.5.5. They had better release it
> anyway after their quality testing and then update inside the normal
> maintenance phase. And based on what oVirt 4.5.5 contains, it could be that
> some of oVirt 4.5.5 features/fixes are already there in OLVM 4.5
> At the momen

[ovirt-users] Oracle Virtualization Manager 4.5 anyone?

[Thomas Hoberg]

Redhat's decision to shut down RHV caught Oracle pretty unprepared, I'd guess, 
who had just shut down their own vSphere clone in favor of a RHV clone a couple 
of years ago.

Oracle is even less vocal about their "Oracle Virtualization" strategy, they 
don't even seem to have a proper naming convention or branding.

But they have been pushing out OV releases without a publicly announced EOL 
almost a year behind Redhat for the last years.

And after a 4.4 release in September 22, a few days ago on December 12th 
actually a release 4.5 was made public.

I've operated oVirt 4.3 with significant quality issues for some years and 
failed to make oVirt 4.4 work with any degree of acceptable stability but 
Oracle's variant of 4.4 proved to be rather better than 4.3 on CentOS7 with no 
noticable bugs, especially in the Hyperconverged setup that I am using with 
GlusterFS.

I assumed that this was because Oracle based their 4.4 in fact on RHV 4.4 and 
not oVirt, but since they're not telling, who knows?

One issue with 4.4 was that Oracle is pushing their UE-Kernel and that created 
immediate issues e.g. with VDO missing modules for UEK and other stuff, but 
that was solved easily enough by using the RHEL kernel.

With 4.5 Oracle obviously can't use RHV 4.5 as a base, because there is no such 
thing with RHV declared EOL and according to Oracle their 4.5 is based on oVirt 
4.5.4, which made the quality of that release somewhat questionable, but 
perhaps they have spent the year that has passed since productively killing 
bugs... only to be caught by surprise again, I presume, by an oVirt release 
4.5.5 on December 1st, that no one saw coming!

Long story slightly shorter, I've been testing Oracle's 4.5 variant a bit and 
it's not without issues.

But much worse, Oracle's variant of oVirt seems to be entirely without any 
community that I could find.

Now oVirt has been a somewhat secret society for years, but compared to what's 
going on with Oracle this forum is teaming with life!

So did I just not look around enough? Is there a secret lair where all those OV 
users are hiding?

Anyhow, here is what I've tested so far and where I'd love to have some 
feedback:

1. Setting up a three node HCI cluster from scratch using OL8.9 and OV 4.5

Since I don't have extra physical hardware for a 3 node HCI I'm using VMware 
workstation 17.5 on a Workstation running Windows 2022, a test platform that 
has been working for all kinds of virtualization tests from VMware ESXi, via 
Xcp-ng and ovirt.

Created three VMs with OL8.9 minimal and then installed OV 4.5. I used the UEK 
default kernels and then had an issue when Ansible is trying to create the 
(local) management engine: the VM simply could not reach the Oracle repo 
servers to install the packages inside the ME. Since that VM is entirely under 
the control of Ansible and no console access of any type is possible in that 
installation phase, I couldn't do diagnostics.

But with 4.4 I used to have similar issues and there switching back to the 
Redhat kernel for the ME (and the hosts) resolved them.

But with 4.5 it seems that UEK has become a baked-in dependency: the OV team 
doesn't even seem to do any testing with the Redhat kernel any more. Or not 
with the HCI setup, which has become deprecated somewhere in oVirt 4.4... Or 
not with the Cockpit wizard, which might be in a totally untested state, or

Doing the same install on OL 8.9 with OV 4.4, however, did work just fine and I 
was even able to update to 4.5 afterwards, which was a nice surprise...

...that I could not repeat on my physical test farm using three Atoms. There 
switching to the UEK kernel on the hosts caused issues, hosts were becoming 
unresponsive, file systems inaccessible, even if they were perfectly fine at 
the Gluster CLI level and in the end the ME VM simply would not longer start. 
Switching back to the Redhat kernel resolved things there.

In short, switching between the Redhat kernel and UEK, which should be 100% 
transparent to all things userland including hypervisors, doesn't work.

But my attempts to go with a clean install of 4.5 on a Redhat kernel or UEK is 
also facing issues. So far the only thing that has worked was a single node HCI 
install using UEK and OV 4.5 and upgrading to OV 4.5 on a virtualized triple 
node OV 4.4 HCI cluster.

Anyone else out there trying these things?

I was mostly determined to move to Proxmox VE, but Oracle's OV 4.5 seemed to be 
handing a bit of a life-line to oVirt and the base architecture is just much 
more powerful (or less manual) than Proxmox, which doesn't have a management 
engine.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 

[ovirt-users] Re: Engine on EL 9

[Thomas Hoberg]

Oracle VM is based on RHV 4.4, which has been declared end of life.

I'm afraid the chances of Oracle taking over oVirt and doing releases on EL9++ 
are slim.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7TTLVE2NIJBA3V2L2LONABMZDCSJN5A/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

Is a change in /etc/pki/vdsm/cert/cacert.pem on the nodes going to 
disrupt the communications between nodes and the engine?


The procedure I followed blew away all of /etc/pki/vdsm on each node.  I 
saved the old one.


Jason

On 8/4/23 14:38, Jason P. Thomas wrote:

I restarted vdsmd and libvirtd after the cert update on each host.

Jason

On 8/4/23 14:34, Derek Atkins wrote:

Did you restart vdsm after updating the certs?
-derek

On Fri, August 4, 2023 2:12 pm, Jason P. Thomas wrote:

I updated the VDSM certs on the hosts and the apache cert on the
engine.  I'm guessing something is wrong with however the engine
interacts with vdsm, I just don't know exactly what to do about it.

Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running. The
libvirt-spice certs had incorrect ownership.  It still is not
connecting
to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General 
SSLEngine

problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:
Now the engine won't start at all and I'm afraid I'm one power 
outage
away from complete disaster.  I need to keep the old location up 
and
functioning for another 4-6 months, so any insights would be 
greatly

appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are
talking
self-hosted engine, then you need to use command like below on host
that
runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf 
list

and hosted-engine --vm-status might be helpful, VM should at least
start
to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and
password
that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
    journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ 


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ 








___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L3HNNMVKBOSHVMZFROSF4JW7PG36GBUQ/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SJM4VP7PYURO77GLIY2POBGBNTE3WMNH/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

cen,
apache.p12 was the first snowflake in this avalanche.  I did find 
something showing how to generate a new one and install it.  That 
actually allowed me to access the engine web interface again.  Kinda 
useless since the engine can't talk to any of the nodes though. Haha.  
Thanks for the info.  I'll look into the engine.p12 between sessions of 
updating my resume.  Haha


Thanks,
Jason

On 8/8/23 17:30, cen wrote:

Hi,

I went through a similar ordeal half a year ago and forgot all the 
exact procedures already but for me, in the end after following all 
the guides and replacing the "standard" certs


it was either engine.p12 or apache.p12 keystore that also had outdated 
certs (apparently mTLS is being used!).


Updating these keystores is not documented anywhere. No idea if you 
are in the same situation but wanted to throw this out there.


Best regards, cen

On 4. 08. 23 20:12, Jason P. Thomas wrote:
I updated the VDSM certs on the hosts and the apache cert on the 
engine.  I'm guessing something is wrong with however the engine 
interacts with vdsm, I just don't know exactly what to do about it.


Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running.  The
libvirt-spice certs had incorrect ownership.  It still is not 
connecting

to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General 
SSLEngine

problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:
Now the engine won't start at all and I'm afraid I'm one power 
outage

away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are 
talking
self-hosted engine, then you need to use command like below on 
host that

runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
and hosted-engine --vm-status might be helpful, VM should at least 
start

to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and 
password

that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
   journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ 


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ 






___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AMVZEWY45QHPEDHJQZGJMZWESN2RZBPB/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UQZ7DE6UWAINN3UJ7OMBW7G2WQ25W2YX/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

cen,
apache.p12 was the first snowflake in this avalanche.  I did find 
something showing how to generate a new one and install it.  That 
actually allowed me to access the engine web interface again.  Kinda 
useless since the engine can't talk to any of the nodes though. Haha.  
Thanks for the info.  I'll look into the engine.p12 between sessions of 
updating my resume.  Haha


Thanks,
Jason

On 8/8/23 17:30, cen wrote:

Hi,

I went through a similar ordeal half a year ago and forgot all the 
exact procedures already but for me, in the end after following all 
the guides and replacing the "standard" certs


it was either engine.p12 or apache.p12 keystore that also had outdated 
certs (apparently mTLS is being used!).


Updating these keystores is not documented anywhere. No idea if you 
are in the same situation but wanted to throw this out there.


Best regards, cen

On 4. 08. 23 20:12, Jason P. Thomas wrote:
I updated the VDSM certs on the hosts and the apache cert on the 
engine.  I'm guessing something is wrong with however the engine 
interacts with vdsm, I just don't know exactly what to do about it.


Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running.  The
libvirt-spice certs had incorrect ownership.  It still is not 
connecting

to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General 
SSLEngine

problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:
Now the engine won't start at all and I'm afraid I'm one power 
outage

away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are 
talking
self-hosted engine, then you need to use command like below on 
host that

runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
and hosted-engine --vm-status might be helpful, VM should at least 
start

to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and 
password

that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
   journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ 


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ 






___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AMVZEWY45QHPEDHJQZGJMZWESN2RZBPB/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WFDTM3I275O4UJKBT6PHFCZAKOUQDIYG/

[ovirt-users] Re: CPU support Xeon E5345

[Thomas Hoberg]

I believe oVirt draws the line at Nehalem, which contained important 
improvements to VM performance like extended page tables. Your Core 2 based 
Xeon is below that line and you'd have to change the code to make it work.

Ultimately oVirt is just using KVM, so if KVM works, oVirt can be made to work, 
too, and KVM still supports much older CPUs.

I've faced similar issues when launching oVirt on Atoms, which are also 
considered below that line, when in fact they support all Nehalem features. The 
problem was that oVirt sets the basic CPU above the line, when it creates the 
self-hosted virtualized management engine, which then fails to start because 
the CPU is below the line. By that time the initial setup VM has already done 
its work, so it's a bit of a nasty surprise and difficult to detect...

I got around the issiue by using a more modern CPU for the initial setup of my 
3 node HCI clusters and then downgraded the CPU baseline afterwards. But in 
theory you could just find the code that sets the CPU type and change it, there 
is a good chance it's hidden away in some Ansible or Python script.

Of course switching systems mid-flight comes with all kinds of other issues, 
but when you're bent on bending the basic requirements the developers have used 
for their code, you need to do the extra work.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FYISK7RMLSCTKJCPNM7IZRZ34D2YU5NA/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

Is a change in /etc/pki/vdsm/cert/cacert.pem on the nodes going to 
disrupt the communications between nodes and the engine?


The procedure I followed blew away all of /etc/pki/vdsm on each node.  I 
saved the old one.


Jason


On 8/4/23 14:38, Jason P. Thomas wrote:

I restarted vdsmd and libvirtd after the cert update on each host.

Jason

On 8/4/23 14:34, Derek Atkins wrote:

Did you restart vdsm after updating the certs?
-derek

On Fri, August 4, 2023 2:12 pm, Jason P. Thomas wrote:

I updated the VDSM certs on the hosts and the apache cert on the
engine.  I'm guessing something is wrong with however the engine
interacts with vdsm, I just don't know exactly what to do about it.

Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running. The
libvirt-spice certs had incorrect ownership.  It still is not
connecting
to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General 
SSLEngine

problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:
Now the engine won't start at all and I'm afraid I'm one power 
outage
away from complete disaster.  I need to keep the old location up 
and
functioning for another 4-6 months, so any insights would be 
greatly

appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are
talking
self-hosted engine, then you need to use command like below on host
that
runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf 
list

and hosted-engine --vm-status might be helpful, VM should at least
start
to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and
password
that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
    journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ 


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ 








___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L3HNNMVKBOSHVMZFROSF4JW7PG36GBUQ/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4FW5ZDRVWFNVF45PBTCSGWQHMX6I2MD4/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

I restarted vdsmd and libvirtd after the cert update on each host.

Jason

On 8/4/23 14:34, Derek Atkins wrote:

Did you restart vdsm after updating the certs?
-derek

On Fri, August 4, 2023 2:12 pm, Jason P. Thomas wrote:

I updated the VDSM certs on the hosts and the apache cert on the
engine.  I'm guessing something is wrong with however the engine
interacts with vdsm, I just don't know exactly what to do about it.

Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running.  The
libvirt-spice certs had incorrect ownership.  It still is not
connecting
to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General SSLEngine
problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:

Now the engine won't start at all and I'm afraid I'm one power outage
away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are
talking
self-hosted engine, then you need to use command like below on host
that
runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
and hosted-engine --vm-status might be helpful, VM should at least
start
to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and
password
that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/






___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L3HNNMVKBOSHVMZFROSF4JW7PG36GBUQ/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

I updated the VDSM certs on the hosts and the apache cert on the 
engine.  I'm guessing something is wrong with however the engine 
interacts with vdsm, I just don't know exactly what to do about it.


Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running.  The
libvirt-spice certs had incorrect ownership.  It still is not connecting
to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General SSLEngine
problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:

Now the engine won't start at all and I'm afraid I'm one power outage
away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are talking
self-hosted engine, then you need to use command like below on host that
runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
and hosted-engine --vm-status might be helpful, VM should at least start
to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and password
that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
   journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/




___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/

[ovirt-users] Re: Certificates expired...

[Jason P. Thomas]

Konstantin,
Right after I sent the email I got the engine running.  The 
libvirt-spice certs had incorrect ownership.  It still is not connecting 
to anything.  Error in Events on the Engine is now: "VDSM 
 command Get Host Capabilities failed: General SSLEngine 
problem"


So status right now is, all VMs are running.  Engine web ui is 
accessible.  Engine shows all hosts as unassigned or Connecting or 
NonResponsive with repeated entries of the above error in Events.


Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:

Now the engine won't start at all and I'm afraid I'm one power outage
away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are talking 
self-hosted engine, then you need to use command like below on host that runs 
ovengine VM (virsh -c 
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list and 
hosted-engine --vm-status might be helpful, VM should at least start to boot in 
order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and password that 
you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
  journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/

[ovirt-users] Certificates expired...

[Jason P. Thomas]

We're moving to a new facility and pretty much building the 
infrastructure out from scratch.  As such, the oVirt 4.4 cluster at our 
old location has floated under notice because it has just worked for 
years.  In July it seems some of the certs expired (specifically the 
engine apache cert) and we just noticed it.  I followed a post for 
changing the apache cert and that allowed us to login to the engine web 
interface, but nothing in the interface showed as connected.  VMs are 
still running, I even rebooted one via ssh before realizing the 
certificate issues.  In "Events" in the engine, it was complaining about 
certs being expired on the hosts.  I found this post to this mailing 
list and followed the instructions possibly in error: 
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NHJNETOIMSHDXMQ6VTW6KS5NEWNBBYKG/ 
Now the engine won't start at all and I'm afraid I'm one power outage 
away from complete disaster.  I need to keep the old location up and 
functioning for another 4-6 months, so any insights would be greatly 
appreciated.


Sincerely,
Jason P. Thomas
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MGDWNQJICB25F7VOGP5EMHGL2KAXVEPC/

[ovirt-users] Re: GPU Passthrough issues with oVirt 4.5

[Thomas Hoberg]

There is little chance you'll get much response here, because it's probably not 
considered an oVirt issue.

It's somewhere between your BIOS, the host kernel and KVM and I'd start by 
breaking it down to passing each GPU separately.

Fromt he PCI-ID it seems to be V100 SMX2 variants that would require a host 
that very likely has a capable and compatible BIOS. I've only ever tried dual 
PCIe V100 in a single VM and that works without any issues on Oracle's RHV 4.4 
variant of oVirt.

So you need to check your BIOS and to ensure that the host kernel isn't 
grabbing any of the GPUs e.g. via Nouveau, perhaps try running a manual KVM VM 
to validate that.

But if you've already solved the problem, it's nice to let people know here.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RNQJ6ZLDH7M5BWL6DDW3TT2ROPFFQXDB/

[ovirt-users] Re: No bootable disk OVA

[Thomas Hoberg]

In my experience OVA exports and imports saw very little QA, even within oVirt 
itself, right up to OVA exports full of zeros on the last 4.3 release (in 
preparation for a migration to 4.4).

The OVA format also shows very little practical interoperatbility, I've tried 
and failed in pretty much every direction between VMware, oVirt/RHV, 
VirtualBox, Xcp-ng and Proxmox.

So transporting the disk images and recreating the machine is certainly the 
more promising option and something that I've used myself, where the guest OS 
was ready to accept that.

Sparse images are another issue, as you have noticed, especially since the 
normal upload/import interfaces love to expand them. So if there is any 
resizing to be done, best do it once the disk image is inside oVirt, not before.

Migrating (handcrated) images has been regarded as being inferior to automated 
builds to such a degree, that QA seems to have completely abandoned any effort 
there, long before oVirt was abandoned itself.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NB7L4OXOYCVZFWZF4BHAR2GOBITFCZAA/

[ovirt-users] Re: Resending with Log: Hosted Engine Deploy Fails at "Waiting for Host to be up"

[Thomas Hoberg]

I have seen this type of behavior when building a HCI cluster on Atoms.

The problem is that at this poing the machine that is generated for the 
management engine has a machine type that is above what is actually supported 
in hardware.

Since it's not the first VM that is run during the setup process, it's not 
really intuitive but the libvirt configuration for that prototypical management 
engine is created by code that assumes to modern a default (I never found the 
source, but since all development has ceased it won't matter any more).

While modern Atoms are actually more like Core CPUs in terms of the instruction 
set they support, my Goldmont Plus/Gemini Lake Atoms were regarded by KVM as 
next to a Nehalem CPU and the VM would refuse to start.

It's very hard to find in the logs, because it is actually a KVM issue (created 
by the oVirt ME creation mechanism, of couse).

I got out of that fix using removable storage: I had the setup run on an 
i7-7700k (was a bit faster, too) and then changed the machine type of the 
management engine (and lowest common denominator for the cluster) to Nehalem 
before transplanting the SSD back into the Atom.

I've gone through that excercise with ovirt 4.3 and again with Oracle's 4.4 
variant, which is by far the most stable oVirt/RHEV available right now.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZKYID23TUFZO7SF3P47MV7MUPZXYC4MZ/

[ovirt-users] Re: VM migration failed after upgrade to 4.4

[Thomas Hoberg]

Live migration across major releases sounds like the sort of feature everybody 
would just love to have but oVirt would support as little as operating clusters 
with mixed release nodes.

AFAIK HCI upgrades from 4.3 to 4.4 were never even described and definitely 
didn't involve live VMs.

I exported all my VMs to an NFS based export domain, redid the HCI from scratch 
and then imported the VMs from the export domain.

And I kept the 4.3 disks around so I could to back if things failed.

The described (non-HCI) upgrade procedures had you up the creek without a 
paddle if things failed half-way...

oVirt was never really enterprise grade.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FIGAL34KPA5QFRGETTL5QMWVOEF5AHCW/

[ovirt-users] Re: oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network

[Thomas Simmons]

It appears that this error is somehow related to the infiniband network. I
found that if I did not assign an IP to the ib0 adapter then
ovirt-hosted-engine-setup would complete successfully. Once the cluster was
up, I could go in and create a new non-vm network, assign gluster and
migration roles, configure the interface and assign the new network to the
ib0 interface. I could then add my other (2) hosts via the UI, however they
would initially come up as NonOperational and the "error: must be number,
not str" would be logged. I could then go into "Setup Host Networks" and
manually assign both networks and successfully activate the hosts.

I'm not sure  if issues will arise during future oVirt host updates,
however everything is working fine at this point.

On Mon, Sep 5, 2022 at 7:18 AM Thomas Simmons  wrote:

> Hello,
> I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
> 8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
> failing when it tries to create the ovirtmgmt network with the error
> "error: Must be number, not str"}]". When this happens, the engine setup
> pauses and if I can login to the bootstrap engine UI and when I attempt to
> manually assign the ovirtmgmt network to the correct nic on the host, I get
> the same error message. This server has (2) active network interfaces - a
> gigabit NIC that will be a VM network for all networks except gluster and
> migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
> gluster and migration (I previously had these servers in the same hardware
> configuration running oVirt 4.3 on CentOS 7 and would like to have the same
> setup again - just with latest versions of EL and oVirt).
>
> I don't believe it's related, however for transparency I should note that
> the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
> because both native EL and elrepo support for my Infiniband HBA was dropped
> in the standard EL8 kernel due to known bugs with that version of the
> kernel. Thanks in advance for any assistance.
>
> Here is the specific error from engine.log on the bootstrap engine. I see
> similar messages in vdsm.log on the host.
>
> 2022-09-04 18:01:10,725-04 INFO
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
> HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
> HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
> vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
> rollbackOnFailure='true', commitOnSuccess='true',
> connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
> bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
> nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
> properties='null', ipv4BootProtocol='STATIC_IP',
> ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
> ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
> ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
> removedNetworks='[]', bonds='[]', removedBonds='[]',
> clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
> 6bc2c376
> 2022-09-04 18:01:10,726-04 INFO
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
> HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
> 2022-09-04 18:01:11,251-04 WARN
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
> value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
> "Attempt to call function:  > with arguments: ({'ovirtmgmt':
> {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
> 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
> 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
> True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
> 2022-09-04 18:01:11,252-04 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
> 'HostSetupNetworksVDS' method
> 2022-09-04 18:01:11,252-04 WARN
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
> value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
> "Attempt to call function:  > with arguments: ({'ovirtmgmt':
> {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',

[ovirt-users] oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network

[Thomas Simmons]

Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).

I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.

Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.

2022-09-04 18:01:10,725-04 INFO
 [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
 [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
 [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: > with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
 [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: > with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: > with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to 

[ovirt-users] Re: Lost space in /var/log

[Michael Thomas]

This can also happen with a misconfigured logrotate config.  If a 
process is writing to a large log file, and logrotate comes along and 
removes it, then the process still has an open filehandle to the large 
file even though you can't see it.  The space won't get removed until 
the process closes the filehandle (eg rebooting).


The following command should give you a list of these ghost files that 
are still open but have been removed from the directory tree:


lsof | grep '(deleted)'

Stackexchange has some additional useful tips:

https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted

--Mike

On 6/22/22 15:49, matthew.st...@fujitsu.com wrote:

Deleted some files to "clean up" /var/log, but the space was not recovered?

Space for deleted files are only recovered when all references to that space 
are removed.   This includes the directory reference, and any open file-handles 
to the file.

The is a popular trick for a self-deleting scratch file.  Create a file, open 
file-handle to it, and then remove the file.   The open file-handle can then be 
written to, and read from, as long as you like, but when the file-handle is 
closed, explicitly or implicitly, the space is automatically recovered.


-Original Message-
From: Andrei Verovski 
Sent: Wednesday, June 22, 2022 3:27 PM
To: users@ovirt.org
Subject: [ovirt-users] Lost space in /var/log

Hi !

I have strange situation with low disk space on /var/log/, yet I can't figure 
out what have consumed so much space.
Look at the du output.

Thanks in advance for any suggestion(s).


# df -h

Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/centos-var   15G  1.3G   13G   9% /var
/dev/mapper/centos-var_log   7.8G  7.1G  293M  97% /var/log
/dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit

#
# du -ah /var/log | sort -n -r | head -n 20

664K/var/log/vdsm
660K/var/log/vdsm/vdsm.log
624K/var/log/gdm
580K/var/log/anaconda/storage.log
480K/var/log/anaconda/packaging.log
380K/var/log/gdm/:0.log.4
316K/var/log/anaconda/syslog
276K/var/log/tuned
252K/var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
220K/var/log/Xorg.0.log
168K/var/log/gdm/:0.log
156K/var/log/yum.log-20191117
156K/var/log/secure-20180726
132K/var/log/libvirt/qemu/NextCloud-Collabora-Active.log
128K/var/log/anaconda/anaconda.log
120K/var/log/hp-snmp-agents
116K/var/log/hp-snmp-agents/cma.log
112K/var/log/vinchin
104K/var/log/vinchin/kvm_backup_service
104K/var/log/tuned/tuned.log.2

#
# find . -printf '%s %p\n'| sort -nr | head -10

16311686 ./rhsm/rhsm.log
4070423 ./cron-20180726
3667146 ./anaconda/journal.log
3409071 ./secure
300 ./rhsm/rhsm.log-20180726
2912670 ./audit/audit.log
1418007 ./sanlock.log
1189580 ./vdsm/vdsm.log
592718 ./anaconda/storage.log
487567 ./anaconda/packaging.log
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JCAQU32UZSEWKYRICOTC6QJMSC5BUZIW/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HKTYSWZ4D3ZLBZ4LYIC4KTUC3NHD6YBS/

[ovirt-users] Re: Emulex OneConnect 10Gb NICs on Ovirt 4.4.10

[Thomas Kamalakis]

Hi Roberto

  We also used the ElRepo drivers in production - most of our e-learning
servers including BBB are on oVirt Gen8 hosts. No problems whatsoever

  The problem is that I could not find drivers for the kernel version of
the 4.4.10 release.

  Thomas


On Sat, 16 Apr 2022, 19:58 Roberto Nunin,  wrote:

> My cent.
>
> I've used both Gen8 and Gen9 with ElRepo drivers, in light production,
> without any problem, more than 2 years.
>
> Also qla2xxx driver was from ElRepo, being cluster with FC storage.
>
>
> Roberto
>
> Il Gio 14 Apr 2022, 08:04 Thomas Kamalakis  ha scritto:
>
>> Thanks Darrell
>>
>>   I think I understand your point, but it looks to me like such a bother
>> to have to do things like this. I will try starting from 4.4.9.
>>
>>   Could a solution be to install Debian and run kvm to install the
>> ovirt-node? I guess you waste some resources doing this but maybe worth it
>> if you can skip all this kernel business.
>>
>>   BR
>>
>>   Thomas
>>
>>
>>
>> On Wed, Apr 13, 2022 at 10:35 PM Darrell Budic 
>> wrote:
>>
>>> I hit this too, RH appears to have limited the supported PCI ids in
>>> their be2net and no longer includes the Emulex OneConnect in the list of
>>> supported IDs. I switched to the EL repo mainline kernels to resolve the
>>> issue on existing systems, it was a bit more fun on systems needing new
>>> installs of 8.x. I think I settled on installing an older system, switching
>>> kernels, and then upgrading to current.
>>>
>>> Good luck,
>>>
>>>   -Darrell
>>>
>>> > On Apr 13, 2022, at 12:12 AM, th...@hua.gr wrote:
>>> >
>>> > Hi
>>> >
>>> >  We have a large number of HP servers where we use Ovirt and we are
>>> having problems upgrading the nodes to Ovirt 4.4.10
>>> >
>>> >  More specifically the servers have the Emulex OneConnect 10Gb NICs
>>> and what we did in previous Ovirt node installations was to download the
>>> driver rpm (e.g. kmod-be2net-12.0.0.0-8.el8_5.elrepo.x86_64.rpm) and then
>>> install it using "rpm -ivh ... " and carrying out "modprobe be2net"
>>> >
>>> >  For 4.18.0-365 we could not find a separate rpm but it looks like
>>> this was included in kernel-modules-4.18.0-365.el8.x86_64.rpm (
>>> https://centos.pkgs.org/8-stream/centos-baseos-x86_64/kernel-modules-4.18.0-365.el8.x86_64.rpm.html)
>>> which is already installed. "modprobe be2net" produces no errors but the
>>> network interfaces do not show up in "ip -a" even after reboot.
>>> >
>>> >  Are we missing something here, or is our infrastructure unusable with
>>> 4.4.10? Things were working for 4.4.9.
>>> >
>>> >  Thanks!
>>> > ___
>>> > Users mailing list -- users@ovirt.org
>>> > To unsubscribe send an email to users-le...@ovirt.org
>>> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>> > oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> > List Archives:
>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/AFSJ76ZGLMM7RVNTNIK7L6PIKG2X2HSV/
>>>
>>>
>>
>> --
>> Dr. Thomas Kamalakis
>> Professor and Dean of the School of Digital Technology
>> Department of Informatics and Telematics, Harokopio University of Athens,
>> Greece
>> Omirou 9, Tavros, Athens, GR17778, Tel: +302109549406,
>> Web: https://galaxy.hua.gr/~thkam, Github:
>> https://github.com/thomaskamalakis/
>>
>> ___
>> Users mailing list -- users@ovirt.org
>> To unsubscribe send an email to users-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/D7LC6BMGWM4DXRDDVZLOJHPEZ5PKGWXB/
>>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NSYMM2DL5F74ELUNAD244NAPMGQCGHNV/

[ovirt-users] Re: Emulex OneConnect 10Gb NICs on Ovirt 4.4.10

[Thomas Kamalakis]

Hi all

  It looks like Gen9 is also not supported in RHEL8. Its a shame, from our
side it looks like a deal breaker since we are not ready to part with our
old but reliable servers.

  Maybe we will try Proxmox.

  Thanks you all for your help

  BR

  Thomas



On Fri, 15 Apr 2022, 00:04 Strahil Nikolov,  wrote:

> I also have Gen8 and those are nearly 8 years old.
> With EL8 Gen8 is no longer supported (this doesn't mean it doesn't
> work)and many of the kernel modules don't recognize the old hardware. As
> recommend, you can use elrepo to solve the problem.
>
> Best Regards,
> Strahil Nikolov
>
> On Thu, Apr 14, 2022 at 19:07, Darrell Budic
>  wrote:
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/5WHKJRN5KBGZ5VIAKDJWYL4LBYVX7CJQ/
>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GEZJXS5GW25M47ZOMHQ2QAMQ553REROO/

[ovirt-users] Re: Emulex OneConnect 10Gb NICs on Ovirt 4.4.10

[Thomas Kamalakis]

ProLiant BL460c Gen8

BR

Thomas

On Thu, Apr 14, 2022 at 5:38 PM Strahil Nikolov 
wrote:

> What gen is your hardware ?
>
> Best Regards,
> Strahil Nikolov
>
> On Thu, Apr 14, 2022 at 9:00, Thomas Kamalakis
>  wrote:
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/D7LC6BMGWM4DXRDDVZLOJHPEZ5PKGWXB/
>
>

-- 
Dr. Thomas Kamalakis
Professor and Dean of the School of Digital Technology
Department of Informatics and Telematics, Harokopio University of Athens,
Greece
Omirou 9, Tavros, Athens, GR17778, Tel: +302109549406,
Web: https://galaxy.hua.gr/~thkam, Github:
https://github.com/thomaskamalakis/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/27KQAW2DB7O25RJPQ5U5RKRRDX4EHJ5B/

[ovirt-users] Re: Emulex OneConnect 10Gb NICs on Ovirt 4.4.10

[Thomas Kamalakis]

Thanks Darrell

  I think I understand your point, but it looks to me like such a bother to
have to do things like this. I will try starting from 4.4.9.

  Could a solution be to install Debian and run kvm to install the
ovirt-node? I guess you waste some resources doing this but maybe worth it
if you can skip all this kernel business.

  BR

  Thomas



On Wed, Apr 13, 2022 at 10:35 PM Darrell Budic 
wrote:

> I hit this too, RH appears to have limited the supported PCI ids in their
> be2net and no longer includes the Emulex OneConnect in the list of
> supported IDs. I switched to the EL repo mainline kernels to resolve the
> issue on existing systems, it was a bit more fun on systems needing new
> installs of 8.x. I think I settled on installing an older system, switching
> kernels, and then upgrading to current.
>
> Good luck,
>
>   -Darrell
>
> > On Apr 13, 2022, at 12:12 AM, th...@hua.gr wrote:
> >
> > Hi
> >
> >  We have a large number of HP servers where we use Ovirt and we are
> having problems upgrading the nodes to Ovirt 4.4.10
> >
> >  More specifically the servers have the Emulex OneConnect 10Gb NICs and
> what we did in previous Ovirt node installations was to download the driver
> rpm (e.g. kmod-be2net-12.0.0.0-8.el8_5.elrepo.x86_64.rpm) and then install
> it using "rpm -ivh ... " and carrying out "modprobe be2net"
> >
> >  For 4.18.0-365 we could not find a separate rpm but it looks like this
> was included in kernel-modules-4.18.0-365.el8.x86_64.rpm (
> https://centos.pkgs.org/8-stream/centos-baseos-x86_64/kernel-modules-4.18.0-365.el8.x86_64.rpm.html)
> which is already installed. "modprobe be2net" produces no errors but the
> network interfaces do not show up in "ip -a" even after reboot.
> >
> >  Are we missing something here, or is our infrastructure unusable with
> 4.4.10? Things were working for 4.4.9.
> >
> >  Thanks!
> > ___
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/AFSJ76ZGLMM7RVNTNIK7L6PIKG2X2HSV/
>
>

-- 
Dr. Thomas Kamalakis
Professor and Dean of the School of Digital Technology
Department of Informatics and Telematics, Harokopio University of Athens,
Greece
Omirou 9, Tavros, Athens, GR17778, Tel: +302109549406,
Web: https://galaxy.hua.gr/~thkam, Github:
https://github.com/thomaskamalakis/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/D7LC6BMGWM4DXRDDVZLOJHPEZ5PKGWXB/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

> On Tue, Feb 22, 2022 at 1:25 PM Thomas Hoberg  k8s does not dictate anything regarding the workload. There is just a
> scheduler which can or can not schedule your workload to nodes.
> 
One of these days I'll have to dig deep and see what it does.

"Scheduling" can encompass quite a few activities and I don't know which of 
them K8 covers.

Batch scheduling (also Singularity/HPC) type scheduling involves also the 
creation (and thus consumption) of RAM/storage/GPUs, so real 
instances/reservations are created, which in the case of pass-through would 
include some hard dependencies.

Normal OS scheduling is mostly CPU, while processes and stroage are alrady 
there and occupy resources and could find an equivalent in traffic steering, 
where the number of nodes that receive traffic is expanded or reduced. K8 to my 
understanding would do the traffic steering as a minimum and then have actions 
for instance creation and deletions.

But given a host with hybrid loads, some with tied resources, others generic 
without: to manage the decisions/allocations properly you need to come up with 
a plan that includes
1. Given a capacity bottleneck on a host, do I ask the lower-layer (DC-OS) to 
create additional containers elsewhere and shut down the local ones or do I 
migrate the running one on a new host?
2. Given a capacity underutilization on a host, how to go best about shutting 
down hosts, that aren't going to be needed for the next hours in a way where 
the migration cost do not exceed the power savings?

To my naive current understanding virt-stacks won't create and shut-down VMs, 
their typical (or only?) load management instrument is VM migration.

Kubernetes (and Docker swarm etc.) won't migrate node instances (nor VMs), but 
create and destory them to manage load.

At large scales (scale out) this swarm approach is obviously better, migration 
creates too much of an overhead.

In the home domain of the virt-stacks (scale in), live migration is perhaps 
necessary, because the application stacks aren't ready to deal with instance 
destruction without service disruption or just because it is rare enough to be 
cheaper than instance re-creation.

In the past it was more clear cut, because there was no live migration support 
for containers. But with CRIU (and its predecessors in OpenVZ), that could be 
done just as seamless as with VMs. And instance creation/deletions were more 
like fail-over scenarios, where (rare) service disruptions were accepted.

Today the two approaches can more easily be mingled but they don't easily mix 
yet, and the negotiating element between them is missing.
> 
> I can understand that a new system like k8s may look intimidating.

Just understanding the two approaches and how they mix is already filling the 
brain capacity I can give them.

Operating that mix is currently quite beyond the fact that it's only a small 
part of my job.
> 
> Best regards,
> Roman
Gleichfalls!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LFNGOUG7BEKA7QSHSO5BCZPPLAT7IUXP/

[ovirt-users] Re: Unable to update self hosted Engine due to missing mirrors

[Thomas Hoberg]

> Le 21/02/2022 à 17:15, Klaas Demter a écrit :
> Thank you, it is ok now but... we
> are faced to the first side effects of 
> an upstream distribution that continuously ships newer packages and 
> finally breaks dependencies (at least repos) into a stable ovirt realease.

which is the effect I was (so pessimistically) pointing to with regards to 
oVirt's future. And I'd say the repos are the least of your worries, it's the 
unavoidable code gaps in a fully agile multiverse that I'm more sceptical about.

It's fully [up]stream or nothing, unless someone outside IBM/Redhat takes that 
fully into their hands.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SOZG67ULC4FQ5CKEMPF33XP4TGRXHJZ5/

[ovirt-users] Re: live storage migration & export speed

[Thomas Hoberg]

This is very cryptic: care to expand a little?

oVirt supports live migration--of VMs, meaning the (smaller) RAM contents--and 
tries to avoid (larger) storage migration.

The speed for VM migration has the network as an upper bound, not sure how 
intelligently unused (ballooned?) RAM is excluded or if there is some type of 
compression/sparsity optimization going on.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HBAW3XCRNFCRTN24AYZ32JAAALRSEF3C/

[ovirt-users] Re: Broke my GlusterFS somehow

[Thomas Hoberg]

I'm glad you made it work!

My main lesson from oVirt from the last two years is: It's not a turnkey 
solution.

Unless you are willing to dive deep and understand how it works (not so easy, 
because there is few up-to-date materials to explain the concepts) *AND* spend 
a significant amount of time to run it through its paces, you may not survive 
the first thing that goes wrong.

And if you do, that may be even worse: Trying to fix a busy production farm 
that has a core problem is stuff for nightmares.

So make sure you have a test environment or two to try things.

And it doesn't have to be physical, especially for the more complex things, 
that require a lot of restart-from-scratch before you do the thing that breaks 
things.

You can run oVirt on oVirt or any other hypervisor that supports nested 
virtualization and play with that before you deal with a "real patient".

I've gone through through catastrophic hardware failures, which would have 
resulted in in a total loss without the resilience oVirt HCI with Gluster 
replicas provided, but I've also lost a lot of hair and nails just running it.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SDBFJNIAMKEH4BSVI6KEHZQKXQX2L4KV/

[ovirt-users] Re: Migrating VMs from 4.3 Ovirt Env to new 4.4 Env

[Thomas Hoberg]

sorry a type there: s/have both ends move/have both ends BOOT the Clonezilla 
ISO...
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OGMGYAXDQ3IO4WGAPFAKWPYQOQGTRMUV/

[ovirt-users] Re: Migrating VMs from 4.3 Ovirt Env to new 4.4 Env

[Thomas Hoberg]

> So as title states I am moving VMs from an old system of ours with alot of 
> issues to a new
> 4.4 HC Gluster envr although it seems I am running into what I have learnt is 
> a 4.3 bug of
> some sort with exporting OVAs. 
The latest release of 4.3 still contained a bug, essentially a race condition 
which resulted in the OVA disk not being exported at all. Unfortunately you 
can't tell from the file size, it's only when you look inside the OVA that 
you'll see the disk image is actually empty.

I hand patched the (single line) fix to make OVA exports work, because they did 
not update 4.3 at that point any more.
https://bugzilla.redhat.com/show_bug.cgi?id=1813028

Since the ability to migrate from one major release to another without such 
problems, I since struggled with the oVirt team's motto "for the entire 
enterprise"
> 
> Some of my OVAs are showing large GB sizes although their actual size may be 
> 20K so they
> are showing no boot device as well, theres really no data there. Some OVAs 
> are fine but
> some are broken. So I was wondering if anyone has dealt with this and whats 
> the best way
> around this issue?
Yes, see above
> 
> I have no access remotely at the moment but when I get to them, I read 
> somewhere its
> better to just detach disks and download them to a HDD for example and build 
> new VMs and
> upload those disks and attach them instead this way? 
You can actually export the disks via the browser, but beware that sparse disks 
may wind up quite big. Unfortunately I relied on VDO/thin to not have me think 
about disk sizes at all (nasty habit picked up from VMware)...

> 
> Please do share if you have any better ways, usually I just Export OVA to an 
> external HDD,
> remount to new ovirt and import OVA but seems a few of these VMs out of alot 
> did not
> succeed and I am running out of downtime window.

While Clonezilla wasn't so great for backup up complete oVirt hosts, which 
might have VDO and Gluster bricks on them, it's turned out a great solution for 
backup up or beaming VMs from one "farm" to another.

E.g. I've used it to move VMs from my oVirt 4.3 HCI cluster to my new XCP-NG 
8.2 HCI cluster or sometimes even to a temporary VMware VM without storing it 
externally: Just create a target VM, have both ends move the Clonezilla ISO and 
start the transfer.

I believe backup domains were new in 4.4 so they only work for 4.4 -> 4.4 
mobility, but export domains, while deprecated, still work on both sides. So 
you can create and fill a backup domain on 4.3, disconnect and reconnect it on 
4.4 and import machines from there.

I believe I've faced some issues with VMs that were too old, or based on 
templates or whatnot, so it's best to try the export/import fully before taking 
down and rebuilding the 4.3, so the VMs in the backup domain can actually be 
recovered.

Generally any VM that is Q35 already on 4.3 seems to be less problematic on 4.4.
> 
> Thanks a bunch, I have noticed repeated names helping me, I am very grateful 
> for your
> help.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6AU3JORW5PC4MWGCMTOE5MNKN2J2TFFS/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

> On Tue, Feb 22, 2022 at 9:48 AM Simone Tiraboschi  wrote:
> 
> 
> Just to clarify the state of things a little: It is not only technically
> there. KubeVirt supports pci passthrough, GPU passthrough and
> SRIOV (including live-migration for SRIOV). I can't say if the OpenShift UI
> can compete with oVirt at this stage.
> 
> 
> Best regards,
> Roman

Well, I guess it's there, mostly because they didn't have to do anything new, 
it's part of KVM/libvirt and more inherited than added.

The main reason I "don't see it coming" is that may create more problems than 
it solves.

To my understanding K8 is all about truly elastic workloads, including 
"mobility" to avoid constraints (including memory overcommit). Mobility in 
quotes, because I don't even know if it migrates containers or just shuts 
instances down in one place and launches them in another: migration itself has 
a significant cost after all.

But if it were to migrate them (e.g. via CRIU for containers and "vMotion" for 
VMs) it would then to also have to understand (via KubeVirt), which devices are 
tied, because they use a device that has too big a state (e.g. a multi-gig CUDA 
workloads), a hard physical dependence (e.g. USB with connected devices) or 
something that could move with the VM (e.g. SR-IOV FC/NIC/INF with a fabric 
that can be re-configured to match or is also virtualized).

A proper negotiation between the not-so-dynamic physically available assets of 
the DC and the much more dynamic resources required by the application are the 
full scope of a virt-stack/k8 hybrid, encompassing a DC/Cloud-OS 
(infrastructure) and K8 (platform) aspects.

While I'd love to have that, I can see how that won't be maintained by anyone 
as a full free-to-use open-souce turn-key solution.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OPAKEXBS4LZSG2HIU3AWGJJCLT22FFGF/

[ovirt-users] Re: Broke my GlusterFS somehow

[Thomas Hoberg]

I think Patrick already gave quite sound advice.

I'd only want to add, that you should strictly separate dealing with Gluster 
and oVirt: the integration isn't strong and oVirt just uses Gluster and won't 
try to fix it intelligently.

Changing hostnames on an existing Gluster is "not supported" I guess, even if I 
understand how that can be a need in real life.

I've had occasion to move HCI clusters from one IP range to another and it's a 
bit like open heart surgery.

I had no control over the DNS in that environment so I made do with /etc/hosts 
on all Gluster members. It's stone age, but it works and you have full control.

So you are basically free to use the old hostnames in /etc/hosts to ensure that 
the gluster pool members are able to talk to each other, while all outside 
access is with the new names. If you can get the management engine to run 
somehow, you can use the trick of using the old aliases in the host file, too, 
to regain operations.

In some cases I've even worked with pure IP addresses for the Gluster setup and 
even that can be all changed in the /var/lib configuration files if and only if 
all Gluster daemons are shut down (they tend to keep their state in memory and 
write it down as the daemon finishes).

Once Gluster itself is happy and "gluster volume status all" is showing "y" on 
all ports and bricks, oVirt generally had no issues at all using the storage. 
It may just take a long time to show things as ok.

The only other piece of advice I can give for a situation like that is to 
decide where your value is and how quickly you need to be back in business.

If it's the VMs running on the infra or the oVirt setup itself?

If it's the VMs and if you're still running on one Gluster leg, I'd concentrate 
on saving the VMs. 

Backup and Export domains on NFS are safe in the sense, that they can typically 
be attached to an oVirt that you rebuilt from scratch, so that's one option. 
OVA exports sometimes work and I've also used Clonezilla to copy VMs across to 
other hypervisors, booting the Clonezilla ISO at both ends and doing a network 
transfer: they lose some attributs and the network may need to be redone 
afterwards, but the data stays safe.

If it's the oVirt setup, I'd rather recommend starting from scratch with the 
latest release and hopefully some backup of the VMs. Fiddling with the database 
is nothing I'd recommend.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KP6YTX2U5OUONAILDIACCA3XMNC7B2YH/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

That's exactly the direction I originally understood oVirt would go, with the 
ability to run VMs and container side-by-side on the bare metal or nested with 
containers inside VMs for stronger resource or security isolation and network 
virtualization. To me it sounded especially attractive with an HCI underpinning 
so you could deploy it also in the field with small 3 node clusters.

But combining all those features evidently comes at too high a cost for all the 
integration and the customer base is either too small or too poor: the cloud 
players are all out on making sure you no longer run any hardware and then it's 
really just about pushing your applications there as cloud native or "IaaS" 
compatible as needed.

E.g. I don't see PCI pass-through coming to kubevirt to enable GPU use, because 
it ties the machine to a specific host and goes against the grain of K8 as I 
understand it.

Memory overcommit is quite funny, really, because it's the same issue as the 
original virtual memory: essentially you lie to your consumer about the 
resources available and then swap pages forth and back in an attempt to make 
all your consumers happy. It was processes for virtual memory, it's VMs now for 
the hypervisor and in both cases it's about the consumer and the provider not 
continously negotiating for the resources they need and the price they are 
willing to pay.

That negotiation is always better at the highest level of abstraction, the 
application itself, which why implementing it at the lower levels (e.g. VMs) 
becomes less useful and needed.

And then there is technology like CXL which essentially turns RAM in to a 
fabric and your local CPU will just get RAM from another piece of hardware when 
your application needs more RAM and is willing to pay the premium something 
will charge for it.

With that type of hardware much of what hypervisors used to do goes into 
DPUs/IPUs and CPUs are just running applications making hypercalls. The kernel 
is just there to bootstrap.

Not sure we'll see that type of hardware at home or in the edge, though...
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PC5SDUMCPUEHQCE6SCMITTQWK5QKGMWT/

[ovirt-users] Re: Xcp-ng, first impressions as an oVirt HCI alternative

[Thomas Hoberg]

>The impression I've got from this mailing list is they are 
intentional design decisions to enforce "correctness" of the cluster.

My understanding of cluster (ever since the VAX) is that it's a fault-tolerance 
mechanism and that was originally one of the major selling points of these 
hypervisor orchestration suites like vSphere, RHV etc. Since that doesn't quite 
work with local storage, I understand why they left it out for live migration. 
AFAIK it works just fine for VMs that are down or disks that are not attached.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YZT66F6VRNNB2EVZDFHMMEKRT7OG4KTY/

[ovirt-users] Re: Xcp-ng, first impressions as an oVirt HCI alternative

[Thomas Hoberg]

> On Tue, Feb 15, 2022 at 8:50 PM Thomas Hoberg  
> For quite some time, ovirt-system-tests did test also HCI, routinely.
> Admittedly, this flow never had the breadth of the "plain" (separate
> storage) flows.

I've known virtualization from the days of the VM/370. And I immediately ran 
and bought the very first VMware [1.0] product, when they did this nice trick 
of using the SMM and binary translation to implement a hypervisor on an 
architecture that excluded VM (except the virtual 8086) [IMHO] on purpose.

I did follow the hypervisor wars, but settled on OpenVZ because it delivered 
compliance, which wasn't firmly established for hypervisors at the time and 
much better consolidation. Redhat then took a very arrogant stance against 
containers at the time, perhaps because Parallels (OpenVZ) and LXC (Canonical) 
were competitors or simply because it had just "won" with Qumranet/KVM against 
Citrix/Xen: I could not convince my Redhat contacts that it wasn't an either/or 
choice, but that both approaches were complimentary. Well SuSE didn't listen 
either, nor did Oracle for that matter.

It took Docker to break that lose and it took Google brushing up their 
let-me-containerise-that into Kubernetes to burn Redhat into action and ditch 
their first OpenShift (sorry, if I don't remember every detail).

Nobody had a perfect grasp of where things were going nor the perfect product 
offer and there was quite a lot of politics involved as well: not every 
decision was based on technical merit.

I came back to VM orchestration because I went from compliance driven 
production architecture to supporting our research labs, which needed GPU 
support for ML and the ability to run set of PoC machines, both stable and 
without (compliance oriented) production support teams. They also needed 
transportable setups, that could be shipped around the world and operated 
without local support.

oVirt offered a turn-key HCI solution with a reasonable minimum of three nodes, 
which could be scaled up or out within one or two orders of computing power 
magnitude. Any single fault could reasonably survive a day or week-end, until a 
spare box could be obtained locally and put in place. Even a double fault 
wouldn't necessarily result in a complete loss or shutdown, just a define 
minimal operations mode, ready for self-recovery, once the hardware was 
replaced: so perfect, in theory, I was positively in love... That love is at 
the root of my current disillusionment, for sure, but it was you who showed the 
"entire enterprise" t*ts!

At the same time it offered the ability grow into something much bigger with 
dedicated storage (and skills), because I understood well enough, that you 
won't run a cloud on HCI. 

oVirt was like a hybrid car and could run on batteries or fuel.

Most importantly it offered running things in a "labs mode" using oVirt and 
CentOS and in a "production mode" using RHEL and RHV, so if one of your 
prototypes got customers, we could just switch it over to our production teams 
who know how to handle auditors looking into every nook and cranny.

So the hybrid car could have air-bags and ESP all around for the Autobahn, or 
sand buggies in the dunes.

These four options are gone now, rather suddenly and with little warning; there 
is only one variant left, which is in full metamorphosis.

It's no longer hybrid, HCI is gone. And there is no longer a choice between a 
compliant safe version and the devil-may-care agile variant, that won't always 
run after every change and close any vulnerability before the cyberwar reaches 
you.

OpenShift with kubevirt may well point much better in the direction of where 
the industry is going, but it's far from the turnkey fault resilient set of 
boxes, which you could just put everywhere and have fixed by people who'd just 
unpack a replacement box and rewire three cables, letting the HCI cluster heal 
itself.

It's also a complete inversion where etcd runs VMs as if they were containers 
while oVirt ran VMs, ...that might have containers in them.

And that may be a good thing to have, for those who want that.
And that even includes me and my company, who is running OpenShift on RHEL in 
the big data centers.

It's just that for the labs and in the field, this doesn't fit, an HCI setup is 
what we need below, while we'll happily run OpenShift and (nested) kubevirt on 
top, to ensure our software is keeping with the times as it evolves.

Now that it's finally in the open that 3/4 oVirt/RHV/HCI options are gone, I'd 
like to help those who like me need one or three of them for their business.

You should help and support that, because you shouldn't want happy oVirt'lers 
to be stranded and potentially angry.

There isn't even an issue of competition any more, because the Xen guys aren't 
selling Kubernetes, just running that, with CoreOS, RHEL or whatnot inside VMs.

> oVirt and Gluster teams did work

[ovirt-users] Re: Xcp-ng, first impressions as an oVirt HCI alternative

[Thomas Hoberg]

Am I pessimistic about the future of oVirt? Quite honestely, yes.

Do I want it to fail? Absolutely not! In fact I wanted it to be a viable and 
reliable product and live up to its motto "designed to manage your entire 
enterprise infrastructure".

It turned out to be very mixed: It has bugs, I consider rather debilitating. It 
has also survived critical failures in hardware, that would have resulted in 
data loss and didn't because Gluster provided replicas that survived.

I have reported on both success and failure. You look through my posts, you 
will find them both.

My impression is that leaders in the oVirt community have not been transparent 
enough about the quality of the support they provide to the various parts. E.g. 
it was only very recently that Nir wrote, that HCI was only ever supported by 
Gluster contributors and not tested by the oVirt core teams.

I believe that the EOL of the downstream products will accelerate the dwindling 
of the communty Sandro has described in his post. I honestly want to be wrong, 
after all I am losing years of work and expertise.

So I posted this report on Xcp-ng, because it may be an options, who like me 
cannot operate on hope alone, but need to provide a service their users can 
trust to have a future without an EOL already formulated.

I would recommend that you all do a proper assessment of both platforms and 
potentially others out there and learn from each other.

With factionism the state of the art cannot progress.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2N2HX2TVY3AQRHWITZSCHHLTYGG2BQFC/

[ovirt-users] Xcp-ng, first impressions as an oVirt HCI alternative

[Thomas Hoberg]

Comments & motivational stuff were moved to the end...

Source/license:
Xen the hypervisor is moved to the Linux foundation. Perpetual open source, 
free to use.
Xcp-ng is a distribution of Xen, produced by a small French company based on 
Xen using (currently) a Linux 4.19 LTS kernel and an EL7 frozen userland from 
July 2020: they promise open source and free to use forever

Mode of operation:
You install Xcp-ng on your (bare metal) hardware. You manage nodes via 
XenOrchestrator, which is a big Node.js application you can run in pretty much 
any way you want.

Business model:
You can buy support for Xcp-ng at different levels of quality.
XenOrchestrator as AN APPLIANCE exposes different levels of functionality 
depending on the level of support you buy.
But you get the full source code of the appliance and can compile it yourself 
to support the full set of qualities.
There is a script out there, which allows you to auto-generate the appliance 
with a single command.
In short you are never forced to pay, but better help can be purchased.

How does it feel to a CentOS/RHEL user?
The userland on the nodes is EL7, but you shouldn't touch that.
CLI is classic Xen, nothing like KVM or oVirt.
I guess libVirt and virsh should be similar, if they live up to their promise 
at all.
Standard user-land on Orchestrator appliance is Debian, but you can build it on 
pretty much any Linux with that script: all interaction is meant to be done via 
Web-UI.

Installation/setup:
There is an image/ISO much like the oVirt node image. Based on a Linux 4.19 LTS 
kernel and an EL7 frozen userland from July 2020 and a freshly maintained Xen 
with tools.
Installation on bare metal or VMs (e.g. for nested experimentation) is a snap, 
HCL isn't extraordinary. I'm still fighting to get the 2.5/5GBit USB3 NIC 
working that I like using for my smallest test systems.

A single command on one node will download the "free"-Orchestrator appliance 
(aka Xoa) and install it as a VM on that node. It's installed as auto-launch 
and just point your brower to its IP to start with the GUI.

There is various other ways to build or run the GUI, which can be run on 
anything remotely Linux, within the nodes or outside: more on this in the next 
section.

The management appliance (Xoa) will run with only 2GB of RAM and 10GB of disk 
for a couple of hosts. You grow to dozens of hosts, give a little more RAM and 
it will be fine. Compare to the oVirt management engine it's very, very light 
seems to have vastly less parts that can break.

And if it does, that doesn't matter, because it is pretty much stateless. E.g. 
pool membership and configuratoin is on the nodes, so if you connect from 
another Xoa they will just carry over. Ditto storage, that configuration which 
oVirt keeps in the management engines Postgres database, is on the nodes in Xcp 
and can be changed by any connected Xoa.

Operation:
Xen nodes are much more autonomous then oVirt hosts. The use whatever storage 
they might have locally, or attached via SAN/NAS/Gluster[!!!] and others.They 
will operate without a management engine much like "single node HCI oVirt" or 
they can be joined into a pool, which opens up live migration and HA. A pool is 
created by telling a node that it's the master now and then adding other nodes 
to join in. The master can be changed and nodes can be moved to other pools. 
Adding and removing nodes to a pool is very quick and easy and it's the same 
for additional storage repositories: Any shared storage added to any node is 
immediately visible to the pool and disks can be flipped between local and 
shared storage very easily (I haven't tried live disk moves, but they could 
work).

Having nodes in a pool qualfies them for live migration (CPU architecture 
caveats apply). If storage is local, it will move with the VM, if storage is 
shared, only RAM will move.

You can also move VMs not sharing a pool and even across different x86 variants 
(e.g. AMD and Intel), when VMs are down. If you've ever daddled with "Export 
domains" or "Backup domains" in oVirt, you just can't believe how quick and 
easy these things are in Xcp-ng. VMs and their disks can be moved, copied, 
cloned, backed-up and restore with a minimum of fuzz including continuous 
backups on running machines.

You can label machines as "HA" so they'll always be restarted elsewhere, should 
a host go down. You can define policies for how to balance workloads across 
hosts and ensure that HA pairs won't share a host, pretty similar to oVirt.

The "free" Xoa has plenty of "upgrade!" buttons all over the place. So I went 
ahead and build an appliance from source, that doesn't have these restrictions, 
just to see what that would get me.

With this script here: https://github.com/ronivay/XenOrchestraInstallerUpdater 
you can build the Xoa on any machine/VM you happen to be running with one of 
the many supported Linux variants.

I build one variant to run as a VM on Xcp-ng and I used another to run 

[ovirt-users] Re: Remove obsolete Gluster hyperconverged doc

[Thomas Hoberg]

> Wait a minute.
> 
> Use of GlusterFS as a storage backend is now deperecated and will be
> removed in a future update?
> 
> What are those who's deployments have GlusterFS as their storage
> backend supposed to use as a replacement?
> 
They are to fully understand the opportunities and risks that come from a 
community open source project:
1. Risk: the project can die at any time from lack of support
2. Opportunity: if you help developing a sufficiently great alternative it may 
be taken aboard

> I'm feeling vibes of the SPICE deprecation all over again. but
> moving all of the VM storage data isn't a quick process, and I don't
> want to move it to something else that will also be depercated by a
> future RH whim

If you want to be safe from "a future RH whim" (it's the community, really, and 
your organization is free to invest more than RH), you need to look for or 
create a community where RH is the minority.

In short, it's somewhat unreasonable to expect others to go to unreasonable 
lenghts to support you for free.

But they could have done a much better job at describing just how shaky things 
were.
> 
> -Patrick Hibbs
> 
> On Fri, 2022-02-04 at 08:42 +0100, Sandro Bonazzola wrote:
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HREYFY7DSVOXTULFUL3KER5BKRYN35XI/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

> On Mon, Feb 7, 2022 at 3:04 PM Sandro Bonazzola  wrote:
> 
> 
> The oVirt storage team never worked on HCI and we don't plan to work on
> it in the future. HCI was designed and maintained by Gluster folks. Our
> contribution for HCI was adding 4k support, enabling usage of VDO.
> 
> Improving on the HCI side is unlikely to come from Red Hat, but nothing
> blocks other companies or contributors from working on this.
> 
> Our focus for 4.5 is Managed Block Storage and incremental backup.
> 
> Nir

Hi Nir,

thank you for the clear message and the confirmation of a suspicion that has 
been growing for a while: HCI is an unwanted stepchild, not an equally 
supported option or even a strategic direction.

And it's perhaps not the only one: in the mean-time I can see VDO and the GUIs 
getting neglected, too.

My problem (and that of potentially many others) is that this unbalanced 
attention wasn't communicated or made visible. HCI may have disappeared from 
the oVirt front page today, in fact it's hard to find these days, but it was 
very prominent on 4.3 when I started.

And as a core developer you may not realize this, but the primary first 
exposure to oVirt by many (most?) newcomers isn't the command line. It's the 
Cockpit wizard, where there are two HCI choices next to the one SAN/NAS option 
where evidently 95% of the oVirt teams work went, which doesn't use Gluster, 
HCI, VDO or any of the GUIs.

I was extremely naive to believe you'd only need to click buttons in GUIs to 
run a 3 node HCI VDO cluster, but actually that expectation came from the 
presentation on this site.

And personally I believe that if that had worked, RHV-HCI would be thriving.

But as it turned out, both the setup GUI and the operational GUI rarely ever 
worked, very likely because both were done by yet another team, while you guys 
only worked and tested at the Ansible level and with SAN/NAS storage.

You will say that oVirt is a community project.

I will say that if you advertise oVirt as "designed to manage your entire 
enterprise infrastructure" and put buttons in GUIs, people will take that at 
face value and expect them to work.

I don't know how many oVirt-HCI deployments I did over the years, but for every 
release I've tried since 4.3.5 or so with the Cockpit HCI wizard, none has ever 
just worked. I've had to dig through logfiles all over the place to fix things 
like blacklisted storage, when I was using Gluster. Then there were VDO options 
that weren't supported yet on EL7 in 4.3 ansible scripts, VDO disappearing 
altogether after a kernel upgrade on EL8, Python 2/3 issues and I don't know 
how many other problems, just to get things set up.

I just did a full fresh set of setups with oVirt 4.4.10 when I was testing the 
compatibility of the various downstream EL8 derivatives and it's still the 
same: the Cockpit setup HCI wizard never just works. By now I know where to 
fiddle, but it saps confidence in the product when every release fails the 
basic setup.

I can hear you saying "our CI only tests at script level", can you guess why 
that has an impact on quality?

And it was the same for nearly half of the operations in the oVirt GUI. I went 
through each and every one of them and for startes I often couldn't find out 
what they were supposed to do, while some even looked downright dangerous to 
click (e.g. "reset brick").

Export and import OVA were pure nightmares, because it turned out that the 
exported machines might in fact contain 100GB of zeros instead of the disk 
image. Even once that was fixed, interoperability with other hypervisors 
(that's the purpose of OVA), was zero.

As explanation I was told here that OVA in-/export wasn't really "meant to be 
used", much like HCI I guess.

There is a cluster upgrade button, but I think I only ever hit it once, only to 
notice that it just created more damage and didn't add convenience. In fact 
upgrading any node became an entirely manual job in the end, because it never 
worked. The gluster daemon never started properly after a reboot and resulted 
in ovirt-ha-broker, ovirt-ha-agent and vdsmd sulking, requiring carefully timed 
restarts to get going again.

And then an upgrade procedure for a high-availability HCI from EL7/oVirt 4.3 to 
EL8/oVirt 4.4 that had 40 steps or so, none of which were allowed to fail and 
with no obvious failback just isn't "enterprise".

To my eyes the value proposition of oVirt was "instant on-premise fault 
tolerant cloud", something I could then use to run VMs or indeed OpenShift on.

oVirt never delivered in an enterprise quality and I can't see it getting any 
closer without a downstream product.

Even a community needs a concrete vision, or perhaps at least a few real use 
cases.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 

[ovirt-users] Re: What happened to oVirt engine-setup?

[Thomas Hoberg]

There I always pictured you two throwing paper balls at each other across the 
office or going for a coffee together...

In the past that difference wouldn't have mattered, I guess.

But with upstream vs downstream your disagreement opens a chasm oVirt can ill 
afford.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6RJ3UXYN57A3V2RRBUFXGVHLB2NGIZGB/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

Sandro, I am ever so glad you're fighting on, buon coraggio!

Yes, please write a blog post on how oVirt could develop without a commercial 
downstream product that pays your salaries.

Ideally you'd add a perspective for current HCI users, many of which chose this 
approach, because a fault-tolerant SAN or NAS wasn't available.

And since most of us here will probably agree that PaaS is the future, another 
post or a section on how a transition to OKD and KubeVirt could be done for 
current oVirt users, would be great, too.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VQQK2ZOCYUZNXX3VKNKDYY4RPY25WV33/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

Alas, Ceph seems to take up an entire brain and mine regularly overflows just 
looking at their home page.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q22UKCGBTUV267DPD66YFDBR3EW4OQNS/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

I just hit across the fact that XOSAN (the "native" HCI solution for XCP-ng) is 
in fact LinStor...

That's what's behind the €6000/year support fee, but there is a beta that's 
community and that I'll try fow now.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/DE4FRDKPS7ZSIB7VZYM6FXHOBO57Z2FQ/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

> Oh i have spent years looking.
> 
> ProxMox is probably the closest option, but has no multi-clustering 
> support. The clusters are more or less isolated from each other, and 
> would need another layer if you needed the ability to migrate between 
> them.
Also been looking at ProxMox for ages. We were using OpenVZ in compliance-heavy 
production so a nice GUI and a VM option seemed ideal. But 
SWsoft/Parallels/Virtuzzo and ProxMox were competing not collaborating and 
diverging with distinct hypervisors and IaaS containers, which seems a silly 
tribal squarrel in face of today's cloud invasion. Never thought LXC might do 
better than OpenVZ (or I might return to Xen from KVM). Redhat fought both IaaS 
containers with nothing but VMs and only got saved by (PaaS) Docker, which they 
then tried to smother with podman and Kubernetes.

But Proxmox is not HCI or only via DIY.
> 
> XCP-ng, cool. No spice support. No UI for managing clustered storage 
> that is open source.
true and the most attractive option seems a paid upgrade (and not ready yet)
Don't think I'd miss SPICE or that it has much of a future.
But nothing HCI has ever deployed so quickly and easily, including vSphere as 
the supposed market leader.
> 
> Harvester, probably the closest / newest contender. Needs a lot more 
> attention / work.
It looks great, thanks for the tip. No idea if they survive the next three 
months.
I am adding the link here, because that name needs to be searched right ;-) 
https://harvesterhci.io/
> 
> OpenNebula, more like a DIY AWS than anything else, but was functional 
> last i played with it.
Lots of material, but is it actually open source? And do they have a free tier? 
They seem to have everything... which makes me more suspicious than happy these 
days.
> 
> 
> Has anyone actually played with OpenShift virtualization (replaces RHV)? 
> Wonder if OKD supports it with a similar model?
The oVirt team had hinted at an integrated container VM solution when I started 
with oVirt. Obviously the pods and etc-daemons need to run somewhere and VMs or 
IaaS containers like OpenVZ or LXC would be a good start. That never 
materialized and evidently they chose to abandon IaaS and HCI completely.

But there is plenty of workloads still out there, that are more comfortable 
with a IaaS abstraction and more concerned with scale-in than scale-out. Or 
which live at the real edge, in the field, on tracks or roads or in the middle 
of an ocean.

I don't quite see how OpenShift replaces RHV, especially not at the [real] 
edge. The software industry may be transitioning towards cloudy application 
models, but it's note quite all there yet.

My impression is that Redhat is very carefully avoiding a CentOS repeat for 
every other open source project they do. Their upstream variants seem far more 
beta in OKD and Kubvirt than oVirt ever was.

"No Free Lunch" seems to have been chiseled into the three letters of their new 
owners for almost a century now.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FUUDMTAAHS3IQGRDWE77SOFC2GLBKJGG/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

> I wonder if Oracle would not be interested in keeping the ovirt.  It will
> really be too bad that ovirt is discontinued.
> 
> https://docs.oracle.com/en/virtualization/oracle-linux-virtualization-man...
> 
> 
> Em sáb., 5 de fev. de 2022 09:43, Thomas Hoberg  escreveu:

I've been looking there, once I discovered they had axed their own original 
product (which used to be the only hypervisor officially sanctioned to get CPU 
partitioning good enough for core based licensing of their SQL servers) and 
gone with oVirt for their commercial offers.

But the most outstanding evidence is that they never made the transition to 
4.4, almost two years after 4.3 support stopped at oVirt. The only news since 
ages is a couple of videos in November: they are up the creek without a paddle, 
too, and that's the only aspect of this EOL that I find slightly amusing.

oVirt is currently made up of so many components over which Redhat has 
exclusive control, that anyone who isn't Redhat's special friend would be crazy 
to take it on, because they couldn't keep things coordinated enough to create a 
product.

Actually, my personal impression is that it's what killed oVirt, especially in 
the HCI variant even inside Redhat.

I've just set up three XCP-ng nodes using nested virtualization on one of my 
home-lab workstations and I've been dumb-struck just how fast and painless it 
went. I then added a Xen-Orchestra appliance(the equivalent of the management 
engine), which again dumbfounded me by just how easy and quick it went (a 
single command grabs the appliance off the Internet and installs it on the 
node).

Of course, then came the inevitable: the nagware! Every other button on the UI 
is nothing but a hint to upgrade to one of the many paid variants. At least one 
of those hidden away buttons actually allows you to upgrade the (freshly 
downloaded..?) nodes, so perhaps the free variant is minimally usable.

The XOSAN HCI variant at €6000/year is definitely out of my home-lab range, but 
might compare favorably to RHV and certainly vSphere or Nutanix. I don't think 
they are quite in the same league, though, but I'll keep checking as much as I 
can.

It's rather ironical that XCP-ng does support Gluster for HCI...

One of the things I loved about oVirt was that you could follow what's going 
on. In parts of our business we have SLAs where outside help will always come 
too late. What I didn't like was that you had to dig deep far too often and 
that it was full of bugs in the setup phase, which had me doubt in its 
operational performance.

In retrospect I have to conceed, that it didn't do so badly there even if I had 
plenty of scares, which is why I still have 4.3 running in the corporate labs: 
until EOL of CentOS7 do us part.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5CASSYL67XUXCECKOJPJ3BSJ45KGQYCW/

[ovirt-users] Re: oVirt alternatives

[Thomas Hoberg]

Xen came before KVM, but ultimately Redhat played a heavy hand to swing much of 
the market but with Citrix it managed to survive (so far).

XCP-ng is a recent open source spin-off, which attempts to gather a larger 
community.
Their XOSAN storage is aimed to deliver a HCI solution somewhat like Gluster. 
But designed only as a companion to XCP-ng, may be easier to maintain and 
faster. v2 is in the works and not backward compatible to v1, so you may want 
to start with other alternatives, of which there are a few.

https://xcp-ng.org/docs/

https://xen-orchestra.com/#!/xo-home
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KXFGINENM6RIO6T5DSCT4QLJQIX5X26B/

[ovirt-users] oVirt alternatives

[Thomas Hoberg]

There is unfortunately no formal announcement on the fate of oVirt, but with 
RHGS and RHV having a known end-of-life, oVirt may well shut down in Q2.

So it's time to hunt for an alternative for those of us to came to oVirt 
because they had already rejected vSAN or Nutanix.

Let's post what we find here in this thread.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YFNNCTW5VVVRKSV2OORQ2UWZ2MTUDD/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

Please have a look here:
https://access.redhat.com/support/policy/updates/rhev/ 

Without a commercial product to pay the vast majority of the developers, there 
is just no chance oVirt can survive (unless you're ready to take over). RHV 4.4 
full support ends this August and that very likely means that oVirt won't 
receive updates past July (judging by how things happened with 4.3).

And those will be CI tested against the Stream Beta not EL8 including RHEL.

Only with a RHV support contract ($) you will receive service until 2024 and 
with extended support ($$$) until 2026.

oVirt is dead already. They have known since October. They should have told us 
last year.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZPQ7DO75CVINFKDWXTSH6D2KM67L5FI4/

[ovirt-users] Re: RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

With Gluster gone, you could still use SAN and NFS storage, just like before 
they tried to compete with Nutanix and vSphere.

Can you imagine IBM sponsoring oVirt, which doesn't make any money without RHV, 
which evidently isn't profitable enough?

Most likely oVirt will lead RHV, in this case to the scrapyard, by months if 
not years.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/T64IKY4DQUPQJXLRRBUCOOJ45FBRXM7Q/

[ovirt-users] RHGS and RHV closing down: could you please put that on the home page?

[Thomas Hoberg]

I just read this message: https://bugzilla.redhat.com/show_bug.cgi?id=2016359

I am shocked but not surprised. And very, very sad.

But I believe this decision needs to be communicated more prominently, as 
people should not get aboard a project already axed.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RMWAAOUJXYPGWCEAHAESV6IHQWIF3CTI/

[ovirt-users] Re: Ignore CPU_TYPE_UNSUPPORTED_IN_THIS_CLUSTER_VERSION ?

[Thomas Hoberg]

Actually the inability to mix CPU vendors is increasingly becoming an issue, 
and probably not just for me.

Of course this isn't an oVirt topic, not even a KVM-only topic, but reaches 
deep into the OS and even applications.

I guess Intel rather likes adding extensions and proprietary registers/flags as 
long as it plays in their favor, but with enough clouds and DCs going to EPYC 
that might flip on them one day.

And in terms of actual instruction support, there should be a rather larger 
intersection of true shared ISA between the two than the last true common 
ancestor (80486?) implies, which could be defined as abstract machine types by 
hypervisor vendors.

The other day I managed to trick a Ryzen 9 via a VMware configuration into 
appearing an Intel CPU well enough to run a Hackintosh that doesn't like AMD 
(CPUs), so it's a definite possibility.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/32SDISBOT6NUQLB6VUCZIPLSMSYXAE6V/

[ovirt-users] Re: Erasure Coding Storage Support

[Thomas Hoberg]

It was this near endless range of possibilities via permutation of the parts 
that originally attracted me to oVirt.

Being clearly a member of the original Lego generation I imagined how you could 
simply add blocks of this and that to rebuild to something new fantastic..., 
limitless gluster scaling and HCI, VDO dedup/compression, SSD caching, nested 
virtualization, geo-replication, OVA support, it just had everything I might 
want!

The truth is rather ugly from what I have gathered around here during almost 
three years.

You don't mention it explicitly, but you evidently are talking about a HCI 
setup, preferably installed, expanded and managed by the nice Cockpit/Engine 
GUIs.

What I learned is that Gluster based HCI receives very little love from the 
oVirt developers, even if it seems to you and me (Lego people?) the most 
attractive option.

My impression is that they tend to work more with the original non-HCI approach 
based on SAN or NFS storage, even if the engine may now by default be a VM, 
when it was separate servers originally.

Gluster, VDO, Ansible, Java engine are all aquisitions, HCI more of a 
management decision to counter Nutanix and I find that following the evolution 
from Moshe Bar's Mosix to Qumranet and the Permabit, Ansible and Gluster 
acquisitions as well as the competitor products helps understand why things are 
as they are.

The current oVirt code base supports single node HCI, which delivers no 
benefits beyond testing. It also supports 3 node HCI, which kind of works with 
the Cockpit wizard (I have never succeeded with an installation where I didn't 
have to do some fiddling).

Beyond that you already branch out into the jungle of not-tested, 
not-supported, even if I remember reading that 6-nodes and 9-node HCI seem 
possible. But quorums with 6 nodes don't seem natural and certainly nobody 
would want to use replicas in a 9-node HCI setup, right?

I've seem actual "not supported" comments in the oVirt Ansible code that stops 
any installation using dispersed volumes, so there is your immediate answer. 
I've tricked it past that point editing Ansible scripts and actually got oVirt 
running with dispersed (erasure coded) volumes on 5 nodes, but it felt too 
wobbly for real use. Instead I've added the CPU/RAM parts of these 5 nodes to a 
3 node HCI setup and then used the disks to create an erasure coded Gluster, 
which then can be used by VMs via Gluster or NFS pretty much like a filer. 

I can't recommend that unless you're willing to pay the price of being on your 
own.

One major issue here is that you can't just create glusters and then merge them 
as any system can only ever be a member of one gluster. And you have to destroy 
volumes before moving to an other gluster: not sure how that rhymes with 
bottleneck-free scalability.

And then there are various parts in oVirt, which look for quorums and find them 
missing even when nodes aren't actually contributing bricks to a volume 
(compute-only hosts or non-HCI volumes).

I guess it's safe to say that everybody here would like to see you trying and 
feeding the changes required to make it work back into the project...

...but it's not "supported out of the box".

P.S. When I'm offered erasure coding, VDO dedup/compression and thin 
allocation, I naturally tend to tick all boxes (originally I also chose SSD 
cache, but quickly changed to SSD-only storage). It's only later when they 
mention somewhere, that you aren't supposed to use them in combination, without 
a full explanation or analysis to follow. Because even at today's SSD storage 
pricing I need all the tricks in the book I stayed with them all, but Gluster 
doesn't seem to be a speed devil, no matter what you put on top.

And to think that there used to be a UDMA/Infiniband option, that later 
disappeared with little comment...

VMs in a 9-node HCI replica gluster could only ever get ~1Gbit/s throughput on 
a 10Gbit/s network, so erasure coding should be the smart choice...
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WQ3LKYCOSKVGYRCCSFW7FW72HHWKUWXB/

[ovirt-users] Re: Installing oVirt on RHEL vs CentOS Stream, Alma or Rocky Linux

[Thomas Hoberg]

> On Tue, Feb 1, 2022 at 7:55 PM Richard W.M. Jones  wrote:
> 
> Would you like to file a doc bug about this?
> 
> oVirt on RHEL is not such a common combination..
Well IBM seems bent on changing that (see the developer license post below)
> 
> In CI we only test on Centos Stream (8, hopefully soon also 9, we'll see),
> and RHV on RHEL.
Which is exactly why BetaStreamOS (upstream to RHEL) based oVirt is breaking 
the value proposition of TrueCentOS (downstream of RHEL) based oVirt as a 
solution "for the entire enterprise".

AFAIK there is no "developer license" of RHV so if you want to run a lab 
environment where the experimentation is done at app or science level (not OS), 
this new beta(OS)-on-beta(VM orchestration) stack just becomes too fragile to 
use, not that oVirt HCI has ever proven better than hardware level fault 
resilience to me, which I believe was its design goal.

> We'll be happy to get success/failure (with details) reports of attempts
> to install/setup both engine and hosts on other OSes.
> 
I have been working on just that, going through Alma, Rocky and Oracle (RHEL, 
Liberty and VzLinux might be added) using a nested VMs on VMware for two 
scenarios:
1. Installing oVirt on a new hardware node (a CentOS8 VM switched to each new 
EL8 brand)
2. Switching a pre-existing oVirt HCI node to a new EL8 brand

In all cases the virtual hosts wind up running a single node HCI gluster, to 
keep things reasonable simple.

All of these worked fine,with some tiny glitches, but that was before the 
gluster8 repos got archived. Currently I am awaiting that being fixed to 
continue testing and hopefully posting a somewhat bigger report here.

But there is another new issue that bothers me quite a bit: The oVirt 
management appliance:

I managed to still get one of the latest CentOS8 based appliances during my 
tests and that could evidently be switched to one of the other EL8 variants, 
once the repo issues are sorted out.

But newer versions of the appliance are built on the UpStreamBeta, which cannot 
(easily?) be switched to a downstream EL8, while the RHEL appliance most likely 
can't be used legally.

Using a Beta engine without the RHEL benefit of vulnerability management is a 
no-go in PCI-DSS or similar compliance heavy environments, so for oVirt to be a 
"solution for the entire enterprise" there needs to be an EL8 based engine, too.

But AFAIK there is not even a publicly available build script for the appliance 
so someone outside RH could build a matching bot. And forced downgrades of the 
engine to one of the EL8 variants might well break the engine...

> Next step is probably for someone to try and build an oVirt node image
> based on a different OS...
I would love trying (if I can find the time...), but I'm not sure where I could 
find build scripts...

One of the reasons I could never use these node images was that the 2.5Gbit USB 
Realtek driver is broken in EL7+EL8, which I used on my Atom based 3 node HCI. 
Building node images would allow adding drivers and putting in nested virt 
support...
> 
> 
> Perhaps I am missing something - is this oVirt-specific, or
> relevant?
> 
> Thanks and best regards,
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ELW5CNTXJNAZ4IS3ONMBDSQ7LFGQNGZI/

[ovirt-users] Re: Installing oVirt on RHEL vs CentOS Stream, Alma or Rocky Linux

[Thomas Hoberg]

> you have 16 developer self support subscriptions from RH, those are more than 
> enough to
> use with ovirt as a cluster/s.

I'd consider that an off-topic post.

And whilst we are off-topic, one of the main attractions of using TrueCentOS 
(the downstream Community ENterprise Operating System) even when my employer 
has a fat Redhat support contract that covers everything I do, is that I just 
didn't have to bother with license management at all.

It's a whole set of processes, activities and knowledge I could simply strike 
off the list and in the context of a lab environment with lots of machines and 
VMs getting created and deleted every day, that is a huge benefit.

Well RHEL not supporting OpenVZ containers was another reason never to bother 
with it and why we ran CentOS userlands even in PCI-DSS production.

I consider the benefit of zero license and subscription management still so 
significant, that I am rather investing into Alma, Rocky, Liberty, OracleLinux 
or VzLinux than dealing with RHEL.

And BetaCentOS aka SomethingStream just isn't an acceptable match for a 
hypervisor or a management engine "for the entire enterprise": Beta is for the 
VMs, if your work is low-level, but 99% of what we do is application level or 
in fact the science above it, where Beta just adds work and risks without any 
benefit.

Sorry for the rant, but posting stuff like that is asking for it.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BTWAULNR4KWV3LLFECBD4EA5LYUA4LVP/

[ovirt-users] Re: ovirt-4.4 morrors failing

[Thomas Hoberg]

> Hi Emilio,
> 
> Yes, looks like the patch that should fix this issue is already here:
> https://github.com/oVirt/ovirt-release/pull/93 , but indeed it still hasn't
> been reviewed and merged yet.
> 
> I hope that we'll have a fixed version very soon, but meanwhile you can try
> to simply apply the changes manually in your *testing* env.

So I did, but I can't help wondering: how well will code tested against 
"stream" work on RHEL, Alma, Rocky, Liberty, VzLinux?
How well will an engine evidently built on "stream" work with hosts based on 
RHEL etc.?
Shouldn't you in fact switch the engine to RHEL etc., too?


> 
> Thanks in advance,
> 
> On Mon, Jan 31, 2022 at 8:05 PM Emilio Del Plato https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MZTD4JSDLSVQ7XBSRCQF4PFRPHJYCVQT/

[ovirt-users] How does the Stream based management engine transition to RHV?

[Thomas Hoberg]

In the recent days, I've been trying to validate the transition from CentOS 8 
to Alma, Rocky, Oracle and perhaps soon Liberty Linux for existing HCI clusters.

I am using nested virtualization on a VMware workstation host, because I 
understand snapshoting and linked clones much better on VMware, even if I've 
tested nested virtualization to some degree with oVirt as well. It makes moving 
forth and back between distros and restarting failed oVirt deployments much 
easier and more reliable than ovirt-hosted-engine-cleanup.

Installing oVirt 4.10 on TrueCentOS systems, which had been freshly switched to 
Alma, Rocky and Oracle went relatively well, apart from Oracle pushing UEK 
kernels, which break VDO (and some Python2 mishaps).

I'm still testing transitioning pre-existing TrueCentOS HCI glusters to Alma, 
Rocky and Oracle.

While that solves the issue of having the hosts running a mature OS which is 
downstream of RHEL, there is still an issue with the management engine being 
based on the upstream stream release: It doesn't have the vulnerability 
managment baked in, which is required even for labs use in an enterprise.

So I'd like to ask our Redhat friends here: How does this work when releases of 
oVirt transition to RHV? Do you backport oVirt changes from Stream to RHEL? 
When bugs are found in that process, are they then fed back into oVirt or into 
the oVirt-to-RHEV proces?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ILSOEBWVAY5J4NRUHS4J76RA5T5TMHAL/

[ovirt-users] Re: Can't upgrade ovirt 4.4.3 to 4.4.9

[Thomas Hoberg]

Unfortunately I have no answer to your problem.

But I'd like to know: where does that leave you?

Are youre severs still running with normal operational tasks performing, are 
you just not able to handle migrations, restarts or is your environment down 
until this gets fixed?

Or were you able to go back to 4.3 and await a fix?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/I6V3U4XJVDP3HNIO7OARAJW4OLGMPPBL/

[ovirt-users] Add static route to ovirt nodes

[Michael Thomas]

I'm running ovirt 4.4.2 on CentOS 8.2. My ovirt nodes have two network 
addresses, ovirtmgmt and a second used for normal routed traffic to the 
cluster and WAN.


After the ovirt nodes were set up, I found that I needed to add an extra 
static route to the cluster interface to allow the hosts to see my ceph 
storage nodes (to make the rbd images visible to the VMs):


10.9.0.0/16 via 10.13.0.1

I can add this route using three different methods:

1) ip route add 10.9.0.0/16 via 10.13.0.1

2) nmcli conn modify enp65s0f0 ipv4.routes "10.9.0.0/16 10.13.0.1"
   nmcli conn down enp65s0f0
   nmcli conn up enp65s0f0

3) vi /etc/sysconfig/network-scripts/route-enp65s0f0
   ifdown enp65s0f0
   ifup enp65s0f0

However, when I reboot the host, the static route goes away.  Methods 2 
and 3 have always given me a persistent static route on other EL8 hosts, 
but not on my ovirt nodes.


What is the correct way to add a persistent static route on an ovirt host?

--Mike
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L4M7MBYH5ZVGMR3W7U2OMVHE7UQJJBW2/

[ovirt-users] Cannot import some OVAs exported from oVirt 4.3.10 into 4.4.9

[Thomas Simmons]

Hello,
I exported all of the VMs on my 4.3.10 cluster to OVAs and while a few have 
imported into my new 4.4.9 cluster just fine, most are failing to import with 
the error below being logged in my engine.log.

Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table 
"vm_static" violates foreign key constraint 
"fk_vm_static_lease_sd_id_storage_domain_static_id"

I haven't been able to find much about this error. Google only has a few hits, 
but it looks like there was a similar issue in 4.2.x that was resolved in one 
of the 4.3 RCs.

https://bugzilla.redhat.com/show_bug.cgi?id=1641703

Any advise to get these VMs imported is appreciated.

More log output:
PL/pgSQL function insertvmstatic(character 
varying,text,integer,integer,integer,integer,uuid,uuid,character 
varying,uuid,timestamp with time 
zone,integer,boolean,boolean,integer,integer,integer,integer,character 
varying,boolean,boolean,boolean,boolean,character 
varying,text,integer,integer,integer,integer,integer,integer,character 
varying,integer,character varying,character varying,character 
varying,integer,character varying,character varying,integer,uuid,character 
varying,boolean,boolean,character varying,boolean,uuid,uuid,uuid,uuid,character 
varying,integer,integer,smallint,character 
varying,boolean,boolean,boolean,uuid,boolean,boolean,boolean,character 
varying,integer,character varying,uuid,uuid,character varying,character 
varying,character varying,uuid,uuid,boolean,boolean,boolean,character 
varying,boolean) line 8 at SQL statement
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:247)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1402)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1065)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1104)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:414)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:374)
at 
org.springframework@5.0.4.RELEASE//org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
at 
org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135)
at 
org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130)
at 
org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76)
at 
org.ovirt.engine.core.dal//org.ovirt.engine.core.dao.DefaultGenericDao.save(DefaultGenericDao.java:93)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmCommandBase.addVmStatic(ImportVmCommandBase.java:628)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmFromExternalProviderCommand.addVmStatic(ImportVmFromExternalProviderCommand.java:309)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmCommandBase.lambda$addVmToDb$8(ImportVmCommandBase.java:560)
at 
org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:181)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmCommandBase.addVmToDb(ImportVmCommandBase.java:559)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmFromExternalProviderCommand.addVmToDb(ImportVmFromExternalProviderCommand.java:391)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.exportimport.ImportVmCommandBase.executeVmCommand(ImportVmCommandBase.java:511)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.VmCommand.executeCommand(VmCommand.java:178)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332)
at 
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010)
at 
org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140)
at 

[ovirt-users] Re: Issue upgrading VDSM in 4.4.9

[Thomas Simmons]

Hello,

I am running into the same issue while attempting a new install of oVirt 4.4 to 
replace my hyperconverged 4.3 cluster. I am running into the issue on one of 
the inital steps (yum/dnf install cockpit-ovirt-dashboard vdsm-gluster 
ovirt-host). After some digging, I found that the version of libvirt that it's 
looking for (libvirt >= 7.6.0-2) is not in the CentOS 8 Advanced Virt repo, but 
is only in the CentOS Stream Advanced Virt repo. 

You can see here that libvirt 7.0.0-14.1 is the latest available in the CentOS 
8 Advanced Virt repo:

http://mirror.centos.org/centos/8/virt/x86_64/advanced-virtualization/Packages/l/

However, the newer libvirt 7.6.0-2 is available in the 8-stream repo:

http://mirror.centos.org/centos/8-stream/virt/x86_64/advancedvirt-common/Packages/l/

I don't know if there will be any ramifications down the road (I'm hoping there 
will not be since Stream tracks just behind the next EL point release), however 
I was able to get past the error by adding the 8-stream Advanced Virt repo to 
my systems by creating the repo file below. It would be nice if a developer 
could respond and advise if this is a "safe" solution, or if this is a known 
problem that they expect to address?

# /etc/yum/repos.d/ovirt-4.4-stream-advanced-virt.repo
[ovirt-centos-stream-advanced-virtualization]
name=Advanced Virtualization CentOS Stream testing packages for $basearch
baseurl=https://buildlogs.centos.org/centos/8-stream/virt/$basearch/advancedvirt-common/
enabled=1
gpgcheck=0
module_hotfixes=1
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FNK5TJW7RWGUIYP5BFKGEDQT3CDX4DXN/

[ovirt-users] Re: Add nodes to single node gluster hyperconverged

[Thomas Hoberg]

Actually quite a few of my 3 node HCI deployments wound up with only the first 
host showing up in oVirt: Neither the hosts nor the gluster nodes were visible 
for nodes #2 and #3.

Now that could be because I am too impatient and self-discovery will eventually 
add them or it could be because I am using sub-part hardware (Atoms and NUCs in 
my home lab), which causes race conditions with Ansible (yes, that does happen 
with oVirt).

And in those cases I wound up installing the two other nodes as an act of 
desperation before restarting the full re-install, either (first) without the 
hosted-engine option (adding it later) or just with the hosted-engine option, 
as if they were ordinary compute(-only) hosts.

It would have the storage volumes change from being described as single to 
replica (even if on the Gluster level they had always been replicas) and 
advance the cluster to a perfectly normal 3-node HCI.

But I have no way of knowing if the 3-node HCI base configuration had been 
injected into the management engine's database already at that point.

If it wasn't, then indeed a 1-9 single step increase should basically work, but 
oVirt still balks at the natural progression from pure replica to erasure code 
volumes, against which there are many objections hard coded into oVirt Ansible 
and library Python code.

I wish I had the time budget and physical hardware to test this, but 
unfortunately I can only hope someone else has.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IYOR6KTYMCOL7ADIDCHJSBODMSGZIY6G/

[ovirt-users] Re: Add nodes to single node gluster hyperconverged

[Thomas Hoberg]

Hi Strahil, I am not as confident as you are, that this is actually what the 
single-node is "designed" for. As a matter of fact, any "design purpose 
statement" for the single-node setup seems missing.

The even more glaring omission is any official guide on how to increase HCI 
from 1 to 9 in steps of one, which for me would really earn the title "HCI".

Anyhow, adding additional nodes for compute is easily done by adding such a 
node in the GUI.

Expanding the Gluster underneath to additional replicas, with arbitration or 
full copies is also easy enough, once you have Gluster experience.

I'd rather doubt the GUI would help you there and what's worse, the GUI doesn't 
easily tell you what it tries to do. By the time you've found and understood 
what it tries from the logfiles, you'd have it done on your own.

Actually I find that it's much easier to replicate the storage setup on any 
additional node via Cockpit. So install/enable cockpit, have a look on the 
machine you're trying to replicate and set up LVM, VDO etc. based on the 
original machine.

Doing the Gluster replicas based on that is much easier. That's also what I've 
used to replace fully failed nodes on 3 node HCI clusters.

Now whether or not oVirt will then treat such a hand-made 1->3 node cluster 
like a 3 node HCI built by itself is something I've never tried.

If I had successfully tested the 3 node to 6 and 9 node expansion, I'd perhaps 
be more confident. But it could just turn out that without fiddling with the 
postgres database in the management engine this won't happen.

But since you can re-install any host to become an management engine host, 
again it may be worth trying (after all we'd all love to know if you made it 
work!).

So Mathieu, bonne chance et j'espère que ça va marcher !
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JTE7UQO27S4K4BBYQ4QE46IEFUNXNSOM/

[ovirt-users] Re: oVirt and the future

[Thomas Hoberg]

Ubuntu support: I feel ready to bet a case of beer, that that won't happen.

oVirt lives in a niche, which doesn't have a lot of growth left.

It's really designed to run VMs on premise, but once you're fully VM and 
containers, cloud seems even more attractive and then why bother with oVirt 
(which has a steep learning curve)?

Of course, you can run oVirt on some clouds to get a bit of your own 
abstraction and mangement layer, but cost vs. benefit are doubtful, especially 
when you can change the code that gave you your infrastructure as code.

I still see some potential where you need a fault tolerant redundant physical 
HCI edge built from small devices like industrial NUCs or Atoms (remote SMB, 
factories, ships, railroads, military/expedition, space stations). But for that 
the quality of the software would have to improve in spades.

If oVirt in HCI was as reliable as CentOS7 on physical hardware, pure software 
update support could perhaps be made to cost no more than the hardware and 
you'd have something really interesting.

But that would require large masses (millions) of deployment to make 
worthwhile, which can't happen with somebody doing a huge amount of initial 
subsidies. And who would be able (and motivated) to shoulder that?

But that's really just my personal opinion, Didi is the much better authority.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/37PPCRBNSMZXI3SYPXWJQHM3WYUPLVLF/

[ovirt-users] Re: oVirt Hosted Engine Offline Deployment

[Thomas Hoberg]

You would do good to mirror everything that oVirt is using, especially if you 
want to install/rebuild while remaining offline.

The 1.1 GB file you mention is the oVirt appliance initial machine image, which 
unfortunately seems to get explicitly deleted from time to time, most likely 
the official clean-up scripts I use, whenever a deployment failed and I want to 
restart from a clean sheet.

If you're operating disconnected, security bugs won't scare you, which is where 
most of the updates are coming from. A good frozen CentOS7 repo state can last 
a long time, but sometimes even those are glitchy so you need to test 
thoroughly before going completely offline.

oVirt 4.3 is *very* stable (nothing done any more), but not free of bugs. You 
may be lucky and in an offline mode the combination may be good... until you 
want online again or add hardware too novel.

It's how I operate currently with most of my oVirt installations: 
CentOS7.latest and oVirt-4.3.last-with-bugs, because I want to test stuff in 
the VMs not underneath the hypervisor (and I am mostly using recycled 
producation hardware to support a lab).
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N2IV4ZR5ZO4XJTMQNFVNRZ3LS7Q7W3SI/

[ovirt-users] Re: Importing VM from Xen Server 7.1

[Thomas Hoberg]

For me this is one of the scenarios where I'd want to use OVA export and import.

Unfortunately a full bidirectional set of tests between oVirt, VMware, Xen 
Server or VirtualBox isn't within oVirt's release pipeline, so very little 
seems to work, not even within oVirt instances.

I think I did get lucky with an oVirt import from a VMware OVA export once, the 
reverse already failed.

Using the KVM/QEMU conversion tools should yield you COW disk images which you 
can upload to oVirt and then add to VMs you define as new. With a bit of luck 
and similar virtual hardware those could perhaps be made bootable. For Windows 
VMs this seems easier these days than with CentOS.

Good luck and please report if you managed, otherwise perhaps redoing the VMs 
on oVirt (and potentially copying data/filesystems afterwards) may be easier.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YR6O2HTY6MTJC6W7D2BNL7ZYY2C4HIKP/

[ovirt-users] Re: Question about PCI storage passthrough for a single guest VM

[Thomas Hoberg]

You're welcome!

My machine learning team members that I am maintaining oVirt for tend to load 
training data is large sequential batches, which means bandwidth is nice to 
have. While I give them local SSD storage on the compute nodes, I also give 
them lots of HDD/VDO based gluster file space, which might do miserably on 
OLTP, but pipes out sequential data at rates at least similar to SATA SSDs with 
a 10Gbit network. Seems to work for them, because to CUDA applications even RAM 
is barely faster the block storage.

PCIe 4.0 NVMe at 8GB/s per device becomes a challenge to any block storage 
abstraction, inside or outside a VM. And when we are talking about NVMe storage 
with "native" KV APIs like FusionIO did back then, PCI pass-through will be a 
necessity, unless somebody comes up with a new hardware abstraction layer.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JTVITH24IOZXOJLQAW6UCQAYHTZWAYUI/

[ovirt-users] Re: Architecture design docs

[Thomas Hoberg]

In the two years that I have been using oVirt, I've been yearning for some nice 
architecture primer myself, but I have not been able to find a nice "textbook 
style" architecture document.

And it does not help that some of the more in-depth information on the oVirt 
site, doesn't seem navigatable from the main "Documentation" link.

This is my very personal opinion, others might have different impressions.

oVirt isn't really a product in the sense that all parts were designed to go 
together. Instead it's a package that has been assembled from quite a few 
rather distinct pieces of technology that Redhat has aquired over the last 
decades. What some might view as proof of extreme flexibility, others will see 
as lack of integration. The oVirt team is careful not to re-implement anything 
on their side, that some other component already delivers. Unfortunately, that 
means you better understand those components underneath, their range of 
functionality and the tooling around them, because oVirt guys won't explain 
what other teams do (e.g. KVM, Gluster, VDO, LVM, Ansible).

KVM originates from Moshe Bar's Qumranet, is the key ingredient of oVirt/RHV 
but also leads a somewhat independent life on its own.

Gluster was a separate scale-out storage company that Redhat aquired, which has 
been passing through its very own trials and tribulations and suffers from lack 
of large scale adoption, especially since scale-out is either cloud or HPC 
where Gluster seems to hold little appeal. I think it's stagnating and its 
level of integration with oVirt is really minimal, even with the tons of work 
developers have done. I consider oVirt's HCI a brilliant value proposition in 
theory and a half baked implementation that one certainly should not use "for 
the entire enterprise".

VDSM is core to oVirt and the philsophical principle AFIK has remained 
unchanged over the last ten years. Its approach also isn't exactly novel (but 
solid!), I see parallels not only with vSphere but right back to things like 
the Tivoli workload scheduler, a mainframe batch scheduler going back decades.

The working principle is to make a deployment plan (~batch schedule), engrave 
that plan into persistent shared storage, so that every worker (host) can 
follow this optimal and conflict free plan, while the manager is free to rest 
or die or be rebooted. It relieves the manager of having to be clustered for 
availability itsself.

VDSM is the agent on every host (or node) responsible for reading and running 
that plan and the engine is the manager, which continously creates the new 
plans.

Originally the manager ran on a separte physical machine, but then somebody 
managed to get that "teleported" into a VM running on the oVirt farm itsself. 
It's a very nice feat and mostly just eliminates the need for a separate host 
to manage the engine. It also allows for an automatic restart of the management 
engine on another host, should the host it ran on fail. But it still needs some 
special treatment compared to any other VM. Again this is something VMware and 
Oracle also managed to do with their VM orchestration tools, perhaps Nutanix 
was the first out of the door with that feature.

Perhaps looking at what the other vendors do, sometimes helps to understand how 
oVirt works, because they do copy ideas from each other (and may be shy about 
documenting that).

There are architecture presentations out there, which unfortunately mostly 
describe the implementation changes made over the years, not the fundamental 
design philosophy nor the current implementation state. That's mostly because 
the implementation has changed fundamentally and keeps changing rapidly so the 
effort to maintain up-to-date docs seems too great. E.g. one of the more recent 
key efforts has been to make Ansible do as much work as possible, where the 
original implementation seems to have used scripts.

But that should not keep someone from doing a textbook on the architecture 
design principles behind oVirt, and perhaps a condensed overview about the 
implementation changes over the years and their motivations.

One could argue that while Redhat is an open source company, it's not an open 
knowledge company. It doesn't necessarily publish all available internal 
documentation and training material they create for their support engineers. 
They do want so sell commercial support.

On the other hand there is conference material, there are lots of RHV related 
Youtube videos scattered around, so you can find a lot of information, just not 
in that tight nice little book you and I seem to wish for.

Unfortunately, I also have not encountered any useful oVirt/RHV architecture 
books. The ones I found were very much "do this, do that" and didn't help me as 
a technical architect.

If I thought that oVirt had a bright and bullish future, I'd be tempted to 
write such a book myself.

With vSphere struggling aginst clouds, I don't see RHV/oVirt doing the right 
things to 

[ovirt-users] Re: import from vmware provider always failis

[Thomas Hoberg]

Honestly, this sounds like a $1000 advice!

Thanks for sharing!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/V6LTN7SHO264NC6HLB6VS2HENL5CT23A/

[ovirt-users] Re: Terrible Disk Performance on Windows 10 VM

[Thomas Hoberg]

You found the issue!

VirtIOSCSI can only do its magic, when it's actually used. And once the boot 
disks was running using AHCI emulation it's a little hard to make it 
"re-attach" to SCSI.

I am pretty sure it could be done, like you could make Windows disks switch 
from IDE to SATA/AHCI with a bit of twiddling in the registry using recovery 
mode.

I think I managed to feed the Windows installer a floppy image with the VirtIO 
drivers at one point, but it took me half a workday I think...

That is one of the reasons I like importing my Windows VMs for oVirt from 
VirtualBox. And it doesn't seem to like the OVAs VMware produces.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4PLHRCBHNXMZBI3MPG53CD23SWXFMQZR/

[ovirt-users] Re: Question about PCI storage passthrough for a single guest VM

[Thomas Hoberg]

You gave some different details in your other post, but here you mention use of 
GPU pass through.

Any pass through will lose you the live migration ability, but unfortunately 
with GPUs, that's just how it is these days: while those could in theory be 
moved when the GPUs were identical (because their amount of state is limited to 
VRAM size), the support code (and kernel interfaces?) simply does not exist 
today.

In that scenario a pass-through storage device won't lose you anything you 
still have.

But you'll have to remember that PCI pass-through works only at the granularity 
of a whole PCI device. That's fine with (an entire) NVMe, because these combine 
"disks" and "controller", not so fine with individual disks on a SATA or SCSI 
controller. And you certainly can't pass through partitions!

It gets to be really fun with cascaded USB and I haven't really tried 
Thunderbolt either (mostly because I have given up on CentOS8/oVirt 4.4)

But generally the VirtIOSCSI interface imposes so little overhead, it only 
becomes noticeable when you run massive amounts of tiny I/O on NVMe. Play with 
the block sizes and the sync flag on your DD tests to see the differences, I've 
had lots of fun (and some disillusions) with that, but mostly with Gluster 
storage over TCP/IP on Ethernet.

If that's really where your bottlenecks are coming from, you may want to look 
at architecture rather than pass-through.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6CJPD6TKL4M44O77RECZYTNVNSSMXJRU/

[ovirt-users] Re: Data recovery from (now unused, but still mounted) Gluster Volume for a single VM

[Thomas Hoberg]

If you manage to export the disk image via the GUI, the result should be a 
qcow2 format file, which you can mount/attach to anything Linux (well, if the 
VM was Linux... it didn't say)

But it's perhaps easier to simply try to attach the disk of the failed VM as a 
secondary to a live VM to recover the data.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SLXLQ4BLQUPBV5355DFFACF6LFJX4MWY/

[ovirt-users] Re: Data recovery from (now unused, but still mounted) Gluster Volume for a single VM

[Thomas Hoberg]

First off, I have very little hope, you'll be able to recover your data working 
at gluster level...

And then there is a lot of information missing between the lines: I guess you 
are using a 3 node HCI setup and were adding new disks (/dev/sdb) on all three 
nodes and trying to move the glusterfs to those new bigger disks?

Resizing/moving/adding or removing disks are "natural" operations for Gluster. 
But oVirt isn't "gluster native" and may not be so forgiving if you just swap 
device paths on bricks.


Practical guides on how to replace the storage without down time (after all 
this is a HA solution, right?) are somehow missing from the oVirt 
documentation, and if I was a rich man, perhaps I'd get myself an RHV support 
contract and see if RHEL engineers would say anything but "not supported".


The first thing I'd recommend is to create some temporary space. I found using 
an extra disk as NFS storage on one of the hosts was a good way to gain some 
maneuvering room e.g. for backups.

You can try to attach the disk of the broken VM as a secondary to another good 
VM to see if the data can be salvaged from there. But before you attach it (and 
perhaps an automatic fsck ruins it for you), you can perhaps create a copy to 
the NFS export/backup (domain).

If you weren't out of space, you'd just create a local copy and work with that. 
You can also try exporting the disk image, but there is a lot of untested or 
slow code in that operation from my experience.

If that image happens to be empty (I've seen that happen) or the data on it 
cannot be recovered, there is little to be gained, by trying to work at the 
GlusterFS level. The logical disk image file will be chunked into 64MB bits and 
their order is buried deep either in GlusterFS or in oVirt and perhaps your 
business is the better place to invest your energy.

But there is a good chance the data portion of that disk image still has your 
data. The fact that oVirt/KVM generally pauses VMs when it has issues with the 
storage, tends to preserve and protect your data rather better than what 
happens when physical hosts suffer brown outs or power glitches.

I guess you'll have learned that oVirt doesn't protect you from making 
mistakes, it only tries to offer some resilience against faults.

It's good and valuable to report these things, because it helps others to 
learn, too.

I sincerely hope you'll make do!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XSCXBSKCI3RKM5FUH57WG6JCMHOE7PMB/

[ovirt-users] Re: Question about pci passthrough for guest (SSD passthrough) ?

[Thomas Hoberg]

> 
> The caveat with local storage is that I can only use the remaining free
> space in /var/ for disk images. The result is the 1TB SSD has around
> 700GB remaining free space.
> 
> So I was wondering about simply passing through the nvme ssd (PCI) to the
> guest, so the guest can utilise the fill SSD.
> 
> Are there any "gotcha's" with doing this other than the usual gpu
> passthrough ones?
> 

The caveat is that you cannot pass through a partial SSD, only a whole device. 
And as a matter of fact, I'd say you can only pass through entire PCI(e) 
devices, so traditional SCSI/SATA disks might not work individually, you'd have 
to pass through the entire controller.

Effectively you'd "unplug" the NVMe from the host and plug it into the guest 
and if that wasn't stopped by some sanity check, neither system would continue 
to run for long, if that is your boot device (or the disk is used otherwise).

I know 3x more IOPS sound attractive, but is your workload really that disk 
bound? Perhaps you'll need to think about some distributed memory cache.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/O32Z2UC7X5AKN6A6Q5HNS77EVYCY3CN6/

[ovirt-users] Re: Error while deploying Hyperconverged oVirt 4.3.3(el7) + GlusterFS

[Thomas Hoberg]

This looks to me like something I've been stumbling across several times...

When trying to redo a filed partial installation of HCI, I often stumbled 
across volume setups not working, even if I had cleared "everything" via the 
'cleanup partial install' button (I don't recall literally what it is called).

As it turns out, the oVirt setup inserts an exclusion filter into 
/etc/lvm/lvm.conf, that blocks any modification to that device as a 
'protective' measure.

Try looking for the gx6iUE-369Z-3FDP-aRUQ-Wur0-1Xhf-v4g79j tag in there and 
eliminate the filter.

Before finding this glitch, I found myself resorting to a full base OS 
reinstall...

And yes, I'd also suggest using the latest 4.3 release, even if that still 
contains bugs.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GU3RUZQ57QOVGKEQ3WBUC6G3CEECNJKQ/

[ovirt-users] Re: Error Adding host to cluster

[Thomas Hoberg]

It's better when you post distinct problems in distinct posts.

I'll answer on the CPU aspect, which may not be related to the networking topic 
at all.

Sounds like you're adding Haswell parts to a farm that was built on Skylakes. 
In order for VMs to remain mobile across hosts, oVirt needs to establish a 
baseline of CPU/instruction set features, that VMs will be let to see, so an 
application/VM launched on a machine with say AVX512 support won't all of a 
sudden find itself on a host that doesn't support that (please note that all 
the various Spectre mitigations resulted in distinct CPU subtypes depending on 
microcode patches being applied to hosts or not).

That's the reason it says in the documentation, that the initial setup should 
be done on the "oldest" machine, to ensure the proper baseline.

In case you add older hardware later, you can lower the base machine type 
(within some limits) by changing the CPU type in the appropriate cluster. 

I do believe it requires restarting the management engine to pick up the change.

Of course, if you want to actually use those nice new instructions, you'll have 
to divide your hosts into distinct clusters.

With regards to the 'ovirt-hosted-engine-setup' playboook being use to add 
hosts, that may just be misleading naming. Rest assured it won't try to add a 
second management engine: that is too complicated to fit into a single playbook.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LKYWXZ6WQ6XWNJKPLK7JL3YHFIH46GZH/

[ovirt-users] Re: oVirt 2021 Spring survey questions

[Thomas Hoberg]

Do you think it would add significant value to your use of oVirt if
- single node HCI could easily promote to 3-node HCI?
- single increments of HCI nodes worked with "sensible solution of quota 
issues"?
- extra HCI nodes (say beyond 6) could easily transition into erasure coding 
for good quota management, distinguishable by volumes?
- oVirt clusters supported easy transition between HCI and SAN/NFS storage as 
initial 1 or 3 node HCI "succeed" into a broader deployment with role 
differentiation?
- it was validated on "edgy hardware" like Atoms, which support 32GB RAM these 
days, nested virtualization with affordable 100% passive hardware?
- oVirt node images were made only from fully validated vertical stacks, 
including all standard deployment variants (SAN/NFS/Gluster 1/3/6/9 node HCI) 
including VDO and all life-cycle operations (updates)?
- import and export of OVA were fully supported/validated standard operations 
against oVirt, VMware and VirtualBox?
- oVirt, Docker, Podman (and OKD) could work side-by-side on hosts, recognizing 
each other's resource allocations and networks instead of each assuming it 
owned the host?
- RealTek drivers, both for onboard and USB3 2.5Gbit were included in the oVirt 
node images and actually worked properly across warm reboots?
- nested virtualization was fully supported with oVirt on oVirt for fully 
testing migration and expansion scenarios before applying them on the physical 
hardware?
- Ansible was just 1x faster?
- oVirt 4.3 could upgrade to 4.4 automagically and with a secure fail-back at 
any point? (ok, I know this is getting madly out of hand...)
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4RZUOERTVUSM2ITRV52E2ST757OLU6H/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

Thank you Gianluca for your honest assessment.

Now if only you'd put that on the home page of oVirt, or better yet, used the 
opportunity to change things.

Yes, after what I know today, I should not have started with oVirt on Gluster, 
but unfortunately HCI is exactly the most attractive grassroots use case and 
the biggest growth opportunity for oVirt.

AFAIK there is no competition out there. If there was, I probably would have 
jumped ship already, as attached (habit dependent) as I am to CentOS.

In theory oVirt could start with single node HCI, and then go to 3nHCI to gain 
resilience. That step is already crucial (ideally witha 2nHCI base on a warm 
storage standby) and currently easy with Gluster, but "not supported" (with no 
warning or explanation that I could remember) by oVirt. Whatever the reason, it 
is not at all obvious to the newcomer, nor should it be unsurmountable to my 
understanding.

And as things go on and you're expanding some clusters, there is a natural 
tendency to go towards dedicated storage say after reaching dual digits on HCI 
nodes, because it offer better price/performance and controls. Again, that 
transition should be "organic" of not seamless, with migration of VMs and their 
disks, perhaps not live, even if gaps there should be closable, too.

Nobody else can do that, not VMware, not Nutanix, nor Proxmox.

And the cloud guys aren't offering anything right there, even if they probably 
should, if only to make sure that any such grass-roots projects can be fork 
lifted easily into their clouds, once their end product takes off.

IMHO this sort of opportunity needs serious seed money, organic growth from the 
community and direct revenues obviously haven't done it yet.

But letting things just go on as they do now, I consider a death knell to oVirt.

Kind regards,

Thomas
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4LMPPCGWK5TPKCO2XNUGSV7V2I7OPHJC/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

and you expect newcomers to find that significant bit of information within the 
reference that you quote as they try to evaluate if oVirt is the right tool for 
the job?

I only found out once I tried to add dispersed volumes to an existing 3 node 
HCI and dug through the log files.

Of course, I eventually managed to remove the nicely commented bits of ansible 
code that prevented adding the volume, only to find that it could not be used 
to run VMs from there or use it for disks.

I can still mount those volumes from inside the VMs via a GlusterFS client and 
I'd guess that there is little if any difference in performance.

For an enterprise HCI solution, the usable intersection between oVirt and 
Gluster is so small, it needs a magnifying glass very top and early in the 
documentation.

Gluster advertises itself as a scale-out file system without any metadata choke 
point as the main differentiator vs. Lustre etc. and with tunable ratio of read 
amplification via replicas and resilience.

Nobody expects scale-out to mean 1 or 3.

With perhaps 6 and 9 as a special option.

Or only replicas actually supported by oVirt, when erasure coding should at 
least in theory give you near perfect scalability, which means you can add 
increments of one or any bigger number and freely allocated between capacity 
and resilience.

It's perfectly legitimate to not support every potential permutation of 
deployment scenarios of oVirt and Gluster.

But the limitations baked in and perhaps even the motivation need to be 
explained from the very start. It doesn't help oVirt's adoption and success if 
people only find out after they have invested heavily under the assumption a 
"scale-out" solution delivers what that term implies.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDWONJCVLEYPWK5OUWPTTJAPVMQQWPTF/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

Hi Strahil,

I've tried to measure the cost or of erasure coding and, more importantly, VDO 
with de-duplication and compression a bit.

Erasure coding should be neglible in terms of CPU power while the vastly more 
complex LZ4 compression (used inside VDO) really is rather impressive at 
1GByte/s single threaded for compression (6Gbyte/s decompression, on a 
25GByte/s memory bus) on the 15Watt NUCs I am using for one cluster.

The storage I/O overhead of erasure coding shouldn't really matter with NVMe 
becoming cheaper than SATA SSD. Perhaps the write amplification needs to be 
watched with SSDs, but a lot of that is writeback tuning and with a Gluster in 
the back, you can commit to RAM as long as you have a quorum (and a UPS).

Actually with Gluster I guess most of the erasure coding would actually be done 
by the client and the network amplification would also be there, but not really 
different between erasure coding and replicas: If you write to nine nodes, you 
write to nine nodes from the client independent of the encoding.

There the ability to say "please continue to use the 4:2 dispersion as I expand 
from 6 to 9 nodes and roll that across on a shard by shard base without me 
having to set up bricks like that", would certainly help.

With all of VDO enabled I get 200MByte/s for a random data workload on FIO via 
Gluster, which becomes 600MByte/s for reads with 3 replicas on the 10Gbit 
network I use, 60% of the theoretical maximum with random I/O.

That's completely adequate, because we're not running HPC or SAP batches here 
and I'd be rather sure that using erasure coding with 6 and 9 nodes won't 
introduce a performance bottleneck, unless I go to 40 or 100GBit on the network.

I'd just really want to be able to choose between say 1, 2 or 3 out of 9 bricks 
being used for redundancy, depending on if it's an HCI block next door, going 
into a ship with months at sea or into a space station.

I'd also probably add an extra node or two to act as warm (even cold) standby 
in critical or hard-to-reach locations, that act as compute-only nodes 
initially (to avoid split quotas), but can be promoted to replace a storage 
node that failed without hands-on intervention.

oVirt HCI is as close at it gets to LEGO computers, but right now it's doing 
LEGO with your hands tied behind your back.

Kind regards, Thomas
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ASJO7WGYNVCZG4CBD3WGLTNERPGAQELO/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

Thank you Gianluca, for supporting my claim: it's patchwork and not "a solution 
designed for the entire enterprise".

Instead it's more of "a set of assets where two major combinations from a 
myriad of potential permutations have received a bit of testing and might be 
useful somewhere in your enterprise".

As such, I see very little future for oVirt as anything that doesn't achieve 
scale these days is doomed to die.

I gather IBM is betting on RH in the cloud, but oVirt isn't designed for that 
(and suffers a license overhead for little if any extra value over the cloud 
native stack) and HCI doesn't make sense in any existing cloud: its mission is 
more like bootstrapping your own. Once you achieve any scale, storage will move 
to specialized appliances.

I can see oVirt and especially the HCI variant on the potentially many 
stopovers from the edge to the cloud core and even in special data center 
holdouts. There the ability to really deliver the best fault resilient 1-9 node 
scalability (somewhat bigger shouldn't be a problem anyway) with the ability to 
carefully tune and mingle between resilience and storage efficiency is key. You 
don't want to redeploy oVirt in a 100 embedded locations across a bigger 
geography, just because you've outgrown 3 nodes.

I could see oVirt run on ships, (including space ships), in factories, in the 
military, train networks, schools or just about any place that need to combine 
some local presence with resilience, flexibility and remote management (but low 
dependence).

But you'd have to go at it strategically and with a truly unified approach 
between the Gluster and oVirt teams.

The management engine, KVM, VDO and Gluster are each brilliant pieces of 
engineering, the combination of which could be a force to reckon with, 
everywhere outside the cloud.

But not with the current approach, where each component is allowed to trudge 
along at its own pace, hopefully not breaking as each is evolving independently.

And of course, the final product must be available free of charge so money 
doesn't get in  the way of scale. When a nation adopts oVirt to digitalize its 
school, or its rail systems, or an industry giant to runs its factories, 
revenue should not be an issue.

And at the low-end you really want to beat QNAP with a 3 node HCI at a similar 
cost and energy footprint e.g. using RASPI modules (or just three last 
generation smartphones for that matter). That's how you'd get the scale.

I hope you'll find something valuable in all this rant!
And sorry for the bother.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4Y6I5CXNCBJJL4ACEWIQTTB7XPXCVPYV/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

> 
> You're welcome to help with oVirt project design and discuss with the
> community the parts that you think should benefit from a re-design.

I consider these pesky little comments part of the discussion, even if I know 
they are not the best style.

But how much is there to discuss, if Redhat has already decided to switch to a 
beta base (CentOS stream) underneath oVirt?

Nobody wants bleeding edge on a hypervisor, except those who develop that 
hypervisor.

oVirt is supposed to deliver a higher reliability than bare metal hardware, by 
providing a fault tolerant design and automatic fault recovery.

But if the software stack that the HA engine builds on is more volatile than 
the hardware below, it simply can't do its work of increasing overall 
resilience: a beta OS kills the value of a HA management stack above.

Only a RHEL downstream CentOS is near solid enough to build on (unless you did 
fully validated oVirt node images). Bleeding edge is what you put inside the 
VMs, not underneath. I don't think I have heard a single oVirt *user* 
advocating the switch to Stream. IMHO it's political and kills oVirt's value 
proposition.

My next major other gripe is that to a newcomer it's not obvious from the start 
that the oVirt 'classic' and the HCI variant are and will most likely remain 
very different beasts, because they have a distinct history and essentially 
incompatible principles.

Classic oVirt started with shared storage, which is always turned on. Theat 
means idle hosts can be turned off and workloads consolidated to minimize 
energy consumption. It aims for the minimal number of hosts to do the job.

Gluster is all about scale out without any choke points, the more hosts the 
better the performance.

And when you combine both in a HCI gluster, turning off hosts requires a much 
better attention as to whether these hosts contribute bricks to volumes in use 
or not.

For a user it's quite natural to mix both, using a set of HCI nodes to provide 
storage and base capacity and then add pure compute nodes to provide dynamic 
workload expansion.

But now I'm pretty sure that's unchartered territory, because I see  terrible 
things happening with quota decisions when computing nodes that don't even 
contribute bricks to volumes are rebooted e.g. during updates.

Making the community responsible for providing a unifying vision, is asking for 
help a bit too late in the game.

And then classic oVirt and HCI oVirt have completely different scaling 
characteristics. Classic will scale from one to N hosts without any issue, but 
HCI won't even go from 1 to 2 or the more sensible 3 nodes. Nor does it then 
allow the transition from replicas to the obviously more attractive dispersion 
volumes as you expand from 3 to 6 or 9 nodes.

How much of a discussion will we have, when I say that I want a Gluster volume 
to expand/grow/shrink and transform from 1 to N bricks and transition between 
replicas, dispersed volumes, sharded or non sharded with oVirt seamlessly 
running on top?

But unfortunately that is the natural expectation any newcomer will have, just 
like I did, when I read all the nice things Redhat had to say about the 
technology.

I hope you won't dispute it's still very much a patchwork and with a very small 
chance of near time resolution.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2CIGCLM5E2CTA2VKDZZOKXGIEQDCO5VT/

[ovirt-users] Re: FreeBSD 13 and virtio

[Thomas Hoberg]

Sigh, please ignore my blabbering about PCI vs PCIe, it seems that the VirtIO 
adapters are all PCI not PCIe independant of the chipset chosen...

In any case I posted the KVM xml configs generated via e-mail to the list and 
they should arrive here shortly.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CXOHJER53M2NORSWXCCVPRLXCJFB4XKE/

[ovirt-users] Re: FreeBSD 13 and virtio

[thomas]

I am attaching both working configs here.

 

Von: Nur Imam Febrianto  
Gesendet: Dienstag, 20. April 2021 14:14
An: Thomas Hoberg ; users@ovirt.org
Betreff: [ovirt-users] Re: FreeBSD 13 and virtio

 

Seems strange. I want to use q35, but whenever I try even to start the
installation (vm disk using virtio-scsi/virtio, net adapter using virtio) it
always shows me the installer doesn't detect any disk. I have an existing VM
too that recently upgraded from 12.2 to 13. It uses i440FX with virtio-scsi
disk and virtio network. If I try to change the machine into q35, it keeps
stuck at boot after "promiscuous mode enabled". 

Don't know what's wrong. Using oVirt 4.4.5. Can you share your VM Config ?

 

From: Thomas Hoberg <mailto:tho...@hoberg.net> 
Sent: 20 April 2021 17:27
To: users@ovirt.org <mailto:users@ovirt.org> 
Subject: [ovirt-users] Re: FreeBSD 13 and virtio

 

q35 with BIOS as that is the cluster default with >4.3.

Running the dmesg messages through my mind as I remember them, the vio
hardware may be all PCIe based, which would explain why this won't work on a
virtual FX 440FX system, because those didn't have PCIe support AFAIK.

Any special reason why you'd want them based on 440FX?

And I also tested with GhostBSD, which is still 12.* based, and that doesn't
seem to have vio support, at least I could not see a hard disk there, which
confirms your observation there.
___
Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
To unsubscribe send an email to users-le...@ovirt.org
<mailto:users-le...@ovirt.org> 
Privacy Statement:
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt
.org%2Fprivacy-policy.html
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovir
t.org%2Fprivacy-policy.htmldata=04%7C01%7C%7C71e5972978224d31def708d903
e6f09b%7C84df9e7fe9f640afb435%7C1%7C0%7C637545112683975628%7CUnk
nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
CI6Mn0%3D%7C1000sdata=effZxEik3q1lfTBn%2BBi4DXIQM6wVJfgiQ7%2Fqr0W1dLM%3
Dreserved=0>
data=04%7C01%7C%7C71e5972978224d31def708d903e6f09b%7C84df9e7fe9f640afb4
35%7C1%7C0%7C637545112683975628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=
effZxEik3q1lfTBn%2BBi4DXIQM6wVJfgiQ7%2Fqr0W1dLM%3Dreserved=0
oVirt Code of Conduct:
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt
.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovir
t.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2Fdata=04%7C01%7C%7C71
e5972978224d31def708d903e6f09b%7C84df9e7fe9f640afb435%7C1%7C0%7C
637545112683975628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=rvQ12IpSElU9D23G7HJvPz%2F
vqqZ0OaUhJ3wI%2BRslIQY%3Dreserved=0>
data=04%7C01%7C%7C71e5972978224d31def708d903e6f09b%7C84df9e7fe9f640afb4
35%7C1%7C0%7C637545112683975628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=
rvQ12IpSElU9D23G7HJvPz%2FvqqZ0OaUhJ3wI%2BRslIQY%3Dreserved=0
List Archives:
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi
rt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2F7ZWDRPXM6G25H4VRE3O
UAH25UIY4JVI3%2F
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ov
irt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2F7ZWDRPXM6G25H4VRE3
OUAH25UIY4JVI3%2Fdata=04%7C01%7C%7C71e5972978224d31def708d903e6f09b%7C8
4df9e7fe9f640afb435%7C1%7C0%7C637545112683975628%7CUnknown%7CTWF
pbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
7C1000sdata=qDf%2BRNi1PXI2cf%2F4XFc6qHCTY6lqOX9lE5Qvyhg4Xls%3Drese
rved=0>
data=04%7C01%7C%7C71e5972978224d31def708d903e6f09b%7C84df9e7fe9f640afb4
35%7C1%7C0%7C637545112683975628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=
qDf%2BRNi1PXI2cf%2F4XFc6qHCTY6lqOX9lE5Qvyhg4Xls%3Dreserved=0

 


  FreeBSD13fx
  779c30b1-3ce1-41e0-be27-3b90b327cd8c
  http://ovirt.org/vm/tune/1.0; xmlns:ovirt-vm="http://ovirt.org/vm/1.0;>

http://ovirt.org/vm/1.0;>
  4194304
  4.5
  False
  false
  2730
  2730
  auto_resume
  1618923190.2439373
  
  
bb2a2f89-685d-4e05-b3fa-d871b15f9d09
6717203c-a594-46e1-951c-ceca54e79e29
77f83726-8d4e-11eb-971d-00163e58a7f8
430fa7fb-3386-4203-b18c-92b0c7a44893
  

  
  16777216
  4194304
  4194304
  64
  1
  

  oVirt
  RHEL
  8.3-1.2011.el8
  143db18a-79e7-d8fa-74ee-1c697a60f24b
  779c30b1-3ce1-41e0-be27-3b90b327cd8c
  oVirt

  
  
hvm

  
  

  
  
Haswell-noTSX

[ovirt-users] Re: FreeBSD 13 and virtio

[Thomas Hoberg]

I tried again with a 440FX chipset and it still worked fine with VirtIO-SCSI 
and the virtual NIC.

I also discovered the other reason I prefer VirtIO-SCSI, which is support for 
discard, always appreciated by SSDs.

It would seem that the virtio family of storage and network adapters support 
both PCI and PCIe incarnations and whoever assembles the final KVM config does 
it right.

Also cross-checked with FreeBSD 12.2 (instead of GhostBSD) and again the 
VirtIO-SCSI disk was not recognized, even if VirtIO support seems to have been 
added with FreeBSD 11.

I'm afraid that I've exhausted my play-time budget with this, but since I used 
to be a fan of BSD (and still love pfSense) it was worth trying.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/D54OBX6NOZPQANF3XXWNPVGVORWB73NG/

[ovirt-users] Re: Cannot delete snapshot

[Thomas Hoberg]

I have used these tools to get rid of snapshots that wouldn't go away any other 
way:
https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/37K5J2X2OUDQKN5J3J7ISOV26FMTHCTY/

[ovirt-users] Re: FreeBSD 13 and virtio

[Thomas Hoberg]

q35 with BIOS as that is the cluster default with >4.3.

Running the dmesg messages through my mind as I remember them, the vio hardware 
may be all PCIe based, which would explain why this won't work on a virtual FX 
440FX system, because those didn't have PCIe support AFAIK.

Any special reason why you'd want them based on 440FX?

And I also tested with GhostBSD, which is still 12.* based, and that doesn't 
seem to have vio support, at least I could not see a hard disk there, which 
confirms your observation there.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7ZWDRPXM6G25H4VRE3OUAH25UIY4JVI3/

[ovirt-users] Re: Is it possible to upgrade 3 node HCI from 4.3 to 4.4?

[Thomas Hoberg]

I'd say very good luck, concentration and coffee...

Would you mind reporting back how it went?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XLU67LHFFOK3WSUOUVSV7IEHXMPQTW5T/

[ovirt-users] Re: intel_iommu=on kvm-intel.nested=1 deactivates rp_filter kernel option

[Thomas Hoberg]

I'd only hazard that the pass-through virtualization settings have zero effect 
on anything network, unless you're actually running a nested VM.

SR-IOV would be an entirely different issue if that is actually used and not 
just enabled.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HHEO6SJXQQMOFQ6OXV43ZWREGZDV7EZY/

[ovirt-users] Re: Two selfhosted engines

[Thomas Hoberg]

This is where a design philosophy chapter in the documentation would really 
help, especially since its brilliance would make for a very nice read.

The self hosted engine (SHE) is in fact extremely highly available, because it 
always leaves behind a fully working 'testament' on what needs to run where 
e.g. in case of a major hickup or servers (including the one running the SHE) 
dying.

And that includes instructions to bring up a new instance of the SHE, which 
will then use this "testament" to create the next one, as workloads and systems 
change.

So as long as there is always a good enough testmament and an SHE running long 
enough to create the next iteration, there is no need for the SHE to run at 
all: the VDSM daemons on each host will faithfully do their work without 
stepping on each other's toes.

The principle isn't really that original to oVirt and has been used for things 
like mainframe job scheduling systems for decades. But it's extremely solid in 
principle as long as the "testament" or execution plan doesn't need to be to 
complex. You can even run a mathematical proof on it then.

On the other hand, two servers will only create chaos, because they'd have to 
decide who is right. That can take so long, the winner might die during the 
negotiations and then what?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FXHRZYFJC4TLOE4GQWDB2EJDJ623XLLX/

[ovirt-users] Re: oVirt Node's Future Regarding CentOS Stream

[Thomas Hoberg]

As long as CentOS was downstream of RHEL, it was a base so solid it might have 
been better than the oVirt node image, even if that was theoretically going 
through some full stack QA testing.

But with CentOS [Up]Stream you get beta quality for the base and then the 
various acquired parts that make up the oVirt house of cards on top.

If then the oVirt node OS were to go through full-stack QA, that could be quite 
the better choice.
But I'm more and more inclined to believe that the if is a false and any 
testing is just unit testing.

Around Christmas VDO got dropped from a kernel update of CentOS8 (still 
non-stream) and my 4.4 oVirt HCI farm dropped dead, until I found out what 
happend.

To me that was the final straw: I will phase oVirt out with CentOS7.
IBM may have a few more paying customers, but oVirt will lose the edge, the 
only place outside the cloud where RedHat can grow without being taxed by the 
cloud service providers.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ASYMLFTIS3YN76Y7OW2BZ6HESW2EA6RA/

[ovirt-users] Re: ova import

[Thomas Hoberg]

The last oVirt 4.3 release contains a bug which will export OVAs with empty 
disks. Just do a du -h  to see if it contains more than the XML 
header and tons of zeros.

Hopefully the orignal VMs are still with you because you'll need to fix the 
python code that does the export: It's a single line of code that needs to be 
added.
https://bugzilla.redhat.com/show_bug.cgi?id=1813028 

It's a bit of a nasty surprise, when you've relied on OVA export/import as the 
most primitve operation for a hypervisor, especially before a migration.

While the bug was fixed for 4.4 in March or earlier, it didn't make it into the 
last 4.3 release... you have the source, after all.

The deprecated export domain seems to work across the 4.3 to 4.4 gap, even if 
they are still deprecated on 4.4 and their handling is nowhere as flexible as 
OVA files.

My understanding of the developer's opinion is that OVA is really just an 
import facility for all those VMware machines you want to migrate to oVirt. 
Nothing you'd ever want to use for anything else, especially not for the other 
direction.

I consider the ability to export an important management VM like a IAS as an 
OVA and be able to run it on say a virtualbox, a potential life saver, 
especially when a farm got lost in a 4.3 to 4.4 migration.

Not that I am planning that any longer...
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CYS3ZR2KIBUMOVH7GELC7M6KCM6DSYMO/

[ovirt-users] Re: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment

[Thomas Hoberg]

It's an effect that also had me puzzled for a long time: To my understanding 
gluster volume command should only ever show peers that contribute bricks to a 
volume, not peers in general.

Now perhaps an exception needs to be made for hosts that have been enabled to 
run the management engine, as that might require Gluster insights.

But I have also seen that with hosts that I only added temporarily to oVirt as 
a compute node and I believe I have even seen really ill effects:

Given G1+G2+G3 holding 3R or 2R+1A bricks for the typical oVirt volumes 
engine/data/vmstore, I had been adding C1+C2+C3 as mere hosts.

When I then rebooted G3, while C1+C2 were also down, all of a sudden G1+G2 
would shut down their bricks (and refuse to bring them back up) for lack of 
quota... I had to bring C1+C2 back up to regain quota and then delete C1+C2 
from oVirt to take them out of the quorum for a volume to which they 
contributed no bricks.

And then often enough, I had to actually detach them as peers via the gluster 
CLI, because the GUI didn't finish the job.

Now of course, that only works when G1+G2+G3 are actually all up, too, because 
otherwise peers can't be detached

I've just posted a query on this issue yesterday: Hopefully someone from the 
development team will shed some insights into the logic, so we can test better 
and potentially open an issue to fix.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HBTFTHXGIKQ7BJPKQLLZF2AVXVHR5M53/

[ovirt-users] Re: How do I share a disk across multiple VMs?

[Thomas Hoberg]

Sharing disks typically requires that you need to coordinate their use above 
the disk.

So did you consider sharing a file system instead?

Members in my team have been using NetApp for their entire career and are quite 
used to sharing files even for databases.

And since Gluster HCI basically builds disks out of a replicated file system, 
why not use that directly? All they do these days is mount some parts of 
oVirt's 'data' volume inside the VMs as a GlusterFS. We just create a separate 
directory to avoid stepping on oVirt's toes and mount that on the clients, who 
won't see or disturb the oVirt images.

They also run persistent Docker storage on these with Gluster mounted by the 
daemon, so none of the Gluster stuff needs to be baked into the Docker images. 
Gives you HA, zero extra copying and very fast live-migrations, which are RAM 
content, only.

I actually added separate Glusters (not managed by oVirt) using erasure coding 
dispersed volumes for things not database, because the storage efficiency is 
much better and a lot of that data is read-mostly. These are machines that are 
seen as pure compute hosts to oVirt, but offer distinct gluster volumes to all 
types of consumers via GlusterFS (NFS or SMB would work, too).

Too bad oVirt breaks with dispersed volumes and Gluster won't support a 
seamless migration from 2+1 replicas+arbiter to say 7:2 dispersed volumes as 
you add tiplets of hosts...

If only oVirt was a product rather than only a patchwork design!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JKRIJDB6PQSIUKA3BEZ4VUERBPSEBR3K/

[ovirt-users] Re: ovirt-hosted-engine-cleanup

[Thomas Hoberg]

ovirt-hosted-engine-cleanup will only operate on the host you run it on.

In a cluster that might have side-effects, but as a rule it will try to undo 
all configuration settings that had a Linux host become an HCI member or just a 
host under oVirt management.

While the GUI will try to do the same with Ansible only, the script seems to do 
a little more and be more thorough. My guess is that it's somewhat of a 
leftover from the older 'all scripts' legacy of oVirt, before they 
re-engineered much of it to fit Ansible.

It's not perfect, however. I have come across things that it would not undo, 
and that were extremely difficult to find (a really clean re-install would then 
help). There was an issue around keys and certificates for KVM that I only 
remember as a nightmare and more recently I had an issue with it's ansible 
scripts inserting an LVM device filter specific for a UID that the HCI setup 
wizard had created, which then precluded it ever working again after running 
the *cleanup scripts.

All I got was "device excluded by a filter" and only much later I found the 
added line in /etc/lvm.conf not undone, which caused this nail-biter.

But generally that message "Is hosted engine setup finished?" mostly indicates 
that the four major daemons for HCI still have issues.

It all starts with glusterd: If you just added a host in HCI with storage, that 
would like to be properly healed. AFAIK that doesn't actually preclude already 
using the machine to host VMs, but in the interest of keeping your head clear, 
I'd suggest concentrating on getting the storage all clean, with gluster 
happily showing connectivity all around (gluster volume status all etc.) and 
'gluster volume heal  [info]' giving good results.

Then you need to check on the services ovirt-ha-broker and ovirt-ha-agent as 
well as vdsmd via 'servicectl  status ' to see what they are 
complaining about. I typically restart them in this order to iron out the 
bumps. As long as you give them a bit of time, restarting them periodically 
doesn't seem to do any harm, but helps them recover from hangs.

And then you simply need patience: Things don't happen immediately in oVirt, 
because few commands ever intervene into anything directly. Instead they set 
flags here and there which then a myriad of interlocking state machine gears 
will pick up and respond to, to do things in the proper order and by avoiding 
the race conditions manual intervention can so easily cause. Even the fastest 
systems won't give results in seconds.

So have a coffee, better yet, some herbal tea, and then look again. There is a 
good chance oVirt will eventually sort it out (but restarting those daemons 
really can help things along, as will looking at those error messages and their 
potential cause).
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C7OV4GPTDVRTTEVE6QFO7TDA4LZIL7XC/

[ovirt-users] Re: Introduction & general question about oVirt

[Thomas Hoberg]

That very much describes my own situation two years ago..., just a slight time 
and geographic offset as my home is near Frankfurt and my work is in Lyon. I 
had been doing 70:1 consolidation via virtualization based on OpenVZ 
(containers, but with a IaaS abstraction), since 2006 because it was zero 
hardware and software budget whereas VMware would have both requried VT-x 
capable hardware and pricey licenses.

OpenVZ turned out extremely easy to use, super reliable and the sysadmin only 
had to learn two or three new commands, not a new abstraction layer. We were 
running payment front office systems, where five minutes of downtime mean a 
very angry boss, one hour of down time costs a yearly bonus and beyond that you 
have to gather your things and go.

That also meant planned 100% availability of eventually consistent data, so 
nothing you could do on a file level: Oracle with streams replication and 
programmed healing or nothing, that meant at the time, can't beat the CAP 
theorem.

(No this isn't a recommendation to go with OpenVZ: That project is about as 
lively as Docker these days).

For a new context with far lighter avaiIability demands but the need to support 
GPUs for machine learning (CUDA breaks OpenVZ containers) I thought that oVirt 
would add managed heterogeneous VMs and hyperconverged storage to the mix, 
again at zero entrance fee with a supported option if that became necessary.

Two years later, I'd say that "I married the wrong woman". It works, but the 
claim "oVirt is an open-source distributed virtualization solution, designed to 
manage your entire enterprise infrastructure" is terribly misleading.

I had planned to get it reasonably stable and ready within weeks of my typical 
stress and failure testing on 1/3 time budget, but it took almost 2 years and 
more like 50% time allocation to learn all the very many things that can go 
wrong, how to diagnose them and how to fix them.

Just yesterday I had one cluster (my 1st functional test 3 node HCI running on 
Atoms, that mostly just sits idle and gets updates, which require reboots), had 
evidently decided to lose one gluster network connection and accumulated 5000+ 
entries in the heal queue during a week of vacation.

It was four hours of careful digging, hundreds of restarted daemons, various 
reboots and a transient situation where the three storage nodes could not see 
or access the storage they were providing, while the management engine ran on a 
compute node and continued to write to a disk that evidently did not exist...  
As a newbie I would have either jumped off a cliff (active users) or tossed the 
project.

However, the magnificent basic design of oVirt had me recover everything 
without a loss... except hair, nerves, general health etc.

And I had to learn the hard way, that just exporting a VM as an OVA and 
expecting it to be importable on any other platform advertising OVA support, or 
even back into oVirt, is not functionality ever included in any regular QA 
testing...

...which might explain why it doesn't work. Or perhaps just no longer, after an 
update.

In short: do not expect anything to work, that you have not fully tested after 
every minor update--several times: everything is extremely raw and ready to 
break at any moment and I can't remember the last time I did a plain vanilla 
install on freshly scrubbed hardware, where I didn't have to help it along 
manually and with digging through dozens of big log files.

What really troubles me: since the basic ingredients for the commercial product 
are the same, I don't see how that might save your bacon. Perhaps 100% 
validated hardware might do it (for a while), but where oVirt is *designed* for 
a maximum of flexibility, it won't reward your taking advantage of that.

I am sticking with it until CentOS7 is end of life, too (just like CentOS8 is 
already), because otherwise I'd have nothing to show for two years of work. But 
if you want to join in, you need to have serious resources to commit. It's most 
likely still smaller than OpenStack, though.

And if you have NetApp filers or SAN, you should not risk HCI. That is super 
elegant as a concept, just like Gluster is a beautiful concept, but very soon 
you'll realize that they were never designed for each other and remain full of 
contradictions.

oVirt may be designed to fit that enterprise role, but in the HCI variant, it 
stil has nowhere near the cohesion and maturity you'd need for that role. 
CentOS, LVM, VDO, KVM, the management engine, Gluster, Ansible are all distinct 
products from what used to be different companies.

And it shows.

Of course, that's just my personal experience and opinion.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: