[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Derek Atkins]

Hi Michal,

On Mon, December 7, 2020 11:43 am, Michal Skrivanek wrote:
>
>
>> On 7 Dec 2020, at 15:35, Gianluca Cecchi 
>> wrote:
>>
>> On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins > > wrote:
>> [snip]
>>
>> The main advantages of ovirt over virt-manager is the access-control and
>> remote-access capabilities.  Specifically, I have several users which
>> have
>> different access to different VMs and their consoles.  Without providing
>> ssh access to the host, I wasn't sure how to provide that access in a
>> clean way via virt-manager.
[snip]
>> +1 here.
>> And I think developers should put more attention in single host
>> environments than lastly done.
>
> well, the truth is it is a corner case. I’m not saying it shouldn’t work
> but as Didi said a single host management was never the main goal. We’ve
> built oVirt around shared storage and DC scalability, that it sort of
> happens to work with single host is….nice, but it’s really not that
> typical. There are better options for desktop-like virtualization in OSS
> world, there’s virt-manager, there’s VM management in cockpit UI,
> gnome-boxes.

As of several years ago, I don't think any of these options worked with
multiple, distributed (remote) users with different capabilities on the
same VM Host.  Has that changed?

>> Derek explained very well what could be many common situations to have a
>> single host environment and the reason not to use virt-manager and such.
>> At time there was the all-in-one and then it was deprecated/abandoned in
>> favour of single host deployment.
>
> yes. but it was never meant to be a real thing in a first place, it was
> created just for demo purposes so it can run on a single laptop.
>
>> Now due to perhaps ansible playbook or new logic in host upgrades it
>> seems to see more and more messages about single host not supported.
>
> it’s not intentional, just not tested enough so it keeps breaking. we
> really can’t test every use case in automation.

I think there are enough users who want this configuration (or, gasp, are
actually using this configuration) that it might warrant a little more
testing.  Yes, we understand that there will be times that we need to shut
down VMs and reboot the system, and those times can be scheduled (like
I've done).  However, that WOULD require a little more support, to at
least have a recipe that works on a single-host hosted-engine solution.

For host cert renewal, that recipe didn't really exist.

[snip]
> I don’t think it would take too much attention, TBH. We’re still dealing
> with 4.4 and el8 complications (it’s still fairly early since GA of a
> major release)

Of course.  (Personally, I think it should have been called 5.0 instead of
4.4, as it requires a full re-install to migrate from 4.3).

> What would make sense, I think, is to identify the actual
> issues/complications and do them differently, like indeed a special local
> playbooks or whatnot, or “special” hacks. And then document on oVirt wiki.
> But otherwise I do not really see them supportable - the amount of work to
> e.g. re-enroll certs on a running host is just too much to do properly,
> and everyone has a different level of “risk” they accept.

Exactly.  Some tested playbook recipes that allows a single-host
hosted-engine deployment to perform these operations is really what I'm
asking for.  Yes, I know it will require rebooting.  But reboot is much
less risky that re-install!

And for the record, after putting the new certificates into place by hand,
just restarting a VM was sufficient to get Spice to pull in the new
cert(s).  So, technically, it LOOKS like I don't have to reboot the whole
system (although I plan to do that tonight) -- I could just shutdown and
re-run each VM.

> HTH,
> michal

Thank you for all your support and everything you do for this project,
Michal.  We very much appreciate it!

-derek

-- 
   Derek Atkins 617-623-3745
   de...@ihtfp.com www.ihtfp.com
   Computer and Internet Security Consultant
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4NVDF5RRX54H5ZP57VIBP4ULECNQF4FJ/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Michal Skrivanek]

> On 7 Dec 2020, at 15:35, Gianluca Cecchi  wrote:
> 
> On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins  > wrote:
> [snip] 
> 
> The main advantages of ovirt over virt-manager is the access-control and
> remote-access capabilities.  Specifically, I have several users which have
> different access to different VMs and their consoles.  Without providing
> ssh access to the host, I wasn't sure how to provide that access in a
> clean way via virt-manager.
> 
> I *used* to use vmware-server, and kept that running as long as I could
> (on a single server).  But that hardware was running long in the tooth, so
> I migrated to ovirt because it gave me a similar web-based UI interface to
> my users so it was relatively easy to migrate them.  When I migrated 4-5
> years ago, virt-manager was still in relative infancy and, IIRC, didn't
> have much remote capability.
> 
> 
> +1 here.
> And I think developers should put more attention in single host environments 
> than lastly done.

well, the truth is it is a corner case. I’m not saying it shouldn’t work but as 
Didi said a single host management was never the main goal. We’ve built oVirt 
around shared storage and DC scalability, that it sort of happens to work with 
single host is….nice, but it’s really not that typical. There are better 
options for desktop-like virtualization in OSS world, there’s virt-manager, 
there’s VM management in cockpit UI, gnome-boxes.

> Derek explained very well what could be many common situations to have a 
> single host environment and the reason not to use virt-manager and such.
> At time there was the all-in-one and then it was deprecated/abandoned in 
> favour of single host deployment.

yes. but it was never meant to be a real thing in a first place, it was created 
just for demo purposes so it can run on a single laptop. 

> Now due to perhaps ansible playbook or new logic in host upgrades it seems to 
> see more and more messages about single host not supported.

it’s not intentional, just not tested enough so it keeps breaking. we really 
can’t test every use case in automation.

> In my opinion to do a reinstall every minor update is not feasible.
> The free version of ESXi could become a more appealing alternative in the 
> near future for these kind of usages, with the risk of potentially eating 
> away also the bigger shape scenario then.
> Perhaps to support again the all-in-one effort could be a good approach.

all-in-one was awful, i’d rather fix those few little things around single host 
deployment:)

> 
> If you think it can get more attention I can open an RFE for revamping 
> all-in-one or an RFE to provide a mechanism to have an host update itself 
> through a locally executed playbook and then "acquired" by the SHE in a 
> second time when exiting global maintenance.
> I don't know the internals enough and I have not the coding skills, but I can 
> support testing the functionality

I don’t think it would take too much attention, TBH. We’re still dealing with 
4.4 and el8 complications (it’s still fairly early since GA of a major release)
What would make sense, I think, is to identify the actual issues/complications 
and do them differently, like indeed a special local playbooks or whatnot, or 
“special” hacks. And then document on oVirt wiki. But otherwise I do not really 
see them supportable - the amount of work to e.g. re-enroll certs on a running 
host is just too much to do properly, and everyone has a different level of 
“risk” they accept.

HTH,
michal

> 
> Thanks for listening
> Gianluca
> 
> 
> 
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/IXXAXY7IMAWSL7MSHHZN5JPHFWEI6GPF/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6UBM6B7IOCPBNOZETEG3N2WTCCPVH4GQ/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Gianluca Cecchi]

On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins  wrote:
[snip]

>
> The main advantages of ovirt over virt-manager is the access-control and
> remote-access capabilities.  Specifically, I have several users which have
> different access to different VMs and their consoles.  Without providing
> ssh access to the host, I wasn't sure how to provide that access in a
> clean way via virt-manager.
>
> I *used* to use vmware-server, and kept that running as long as I could
> (on a single server).  But that hardware was running long in the tooth, so
> I migrated to ovirt because it gave me a similar web-based UI interface to
> my users so it was relatively easy to migrate them.  When I migrated 4-5
> years ago, virt-manager was still in relative infancy and, IIRC, didn't
> have much remote capability.
>
>
+1 here.
And I think developers should put more attention in single host
environments than lastly done.
Derek explained very well what could be many common situations to have a
single host environment and the reason not to use virt-manager and such.
At time there was the all-in-one and then it was deprecated/abandoned in
favour of single host deployment.
Now due to perhaps ansible playbook or new logic in host upgrades it seems
to see more and more messages about single host not supported.
In my opinion to do a reinstall every minor update is not feasible.
The free version of ESXi could become a more appealing alternative in the
near future for these kind of usages, with the risk of potentially eating
away also the bigger shape scenario then.
Perhaps to support again the all-in-one effort could be a good approach.

If you think it can get more attention I can open an RFE for revamping
all-in-one or an RFE to provide a mechanism to have an host update itself
through a locally executed playbook and then "acquired" by the SHE in a
second time when exiting global maintenance.
I don't know the internals enough and I have not the coding skills, but I
can support testing the functionality

Thanks for listening
Gianluca
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IXXAXY7IMAWSL7MSHHZN5JPHFWEI6GPF/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Derek Atkins]

Hi Didi,

Sorry for the multiple emails yesterday.  I'm going to respond to all of
your responses in this one.

On Mon, December 7, 2020 3:31 am, Yedidyah Bar David wrote:
> On Sun, Dec 6, 2020 at 8:14 PM Derek Atkins  wrote:
>>
>> Hi again,
>>
>> I also noticed that ca.pem was not updated -- it's still using Sha1.
>
> You are right - we didn't make engine-setup recreate existing certs
> for this - "Renew" deals with other stuff [1]. We only change the
> default for new ones [2], and wrote a procedure [3][4] for doing this
> manually. At the time, this wasn't mandatory - browsers didn't reject
> sha1. Perhaps now it should be.

I should point out that it's not the browsers that are rejecting SHA1, but
it is remote-viewer that is.  My Fedora-33 firefox connected to my
Sha1-using Ovirt HTTPS just fine, without any complaints.  Granted, as I
note later, these certs were already imported and accepted in firefox
years ago, so that could be why there was not complaints.

However, the console.vv file sent to remote-viewer includes the CA cert --
but I'm not sure if it's complaining about that or the host cert that gets
sent during the connection; I can't tell from the output.  I know it's the
CA cert that's sent in the .vv file, but I'm not 100% sure which
particular source is being used, and I'm not sure which cert is considered
"bad" by the viewer.

[snip]
>> I don't know if this will be an issue with remote-viewer if I wind up
>> refreshing the host cert?
>
> As I said, at the time it didn't seem to be mandatory, and docs seemed
> to be enough. If you feel otherwise, please open a bug.

I already refreshed the CA cert, so unfortunately I wont be able to test
this for sure.  I know that refreshing the CA cert on the engine alone is
not sufficient -- the console.vv file still has the old one, even after an
engine restart.

> I think there is a difference, or at least there was, between what
> browsers
> did/do with https certs, and what they did with CA certs.

Probably true, but firefox was not complaining with the Sha1 certs.

> If you had a CA cert already accepted/imported/trusted by the browser,
> and then you entered a site with a cert signed by this CA, but with a
> SHA1 signature, this was one separate case. Browsers started warning/
> rejecting them earlier.

I think I had already accepted the site cert which is probably why it
wasn't complaining about it being SHA1.

> If you have a CA cert with a SHA1 signature, and want to import that to
> a browser, that's another case. I didn't test recently (or much over time,
> other than working on these bugs) with recent browsers, but I think it
> took longer until they rejected (if indeed they do - not sure all of them
> do).

Indeed.  I already had both the SHA1 CA cert and the SHA1 host cert
accepted in my Firefox trust before I upgraded, so perhaps that's why I
didn't see any issues in F33.  I removed both and re-imported the (new) CA
cert.

>> One more question:
>>
>> Can you verify that etc/pki/libvirt/clientcert.pem,
>> etc/pki/vdsm/certs/vdsmcert.pem, and
>> etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same
>> certificate (on the host)?  By a quick find | grep all three of these
>> files appear to be the .cer certificate file?
>
> Yes, and also vdsm/libvirt-vnc/server-cert.pem .

I don't see this directly in /etc/pki on the host?  All I see is:
# ls -l /etc/pki/vdsm/
total 8
drwxr-xr-x. 2 vdsm kvm 4096 Dec  6 17:16 certs
drwxr-xr-x. 2 vdsm kvm   80 Jun  7  2020 keys
drwxr-xr-x. 2 vdsm kvm 4096 Dec  6 17:18 libvirt-spice

And /etc/pki/vdsm does not exist on the engine.  Indeed:

# find /etc/pki -name server-cert.pem
/etc/pki/vdsm/libvirt-spice/server-cert.pem


>> Does it matter that ca.der didn't change?  I don't know if that is a
>> self-signed cert that might be problematic?
>
> ca.der is not used by anything, you can ignore it. The private key of
> the CA is in /etc/pki/ovirt-engine/private/ca.pem, and the public key
> is in /etc/pki/ovirt-engine/ca.pem. That's what all tools use.

Actually, I verified that ca.der *IS* used -- that's what gets sent out if
you access
http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate=X509-PEM-CA
-- so I had to update that in order to make the new cert available.



> Generally speaking, the project considers the "standard" use case to be a
> setup of at least two hosts, and at least one host "extra" (in terms of
> capacity), so that if a host fails, you can still keep everything up. In
> that regard, a single-host setup is considered a kind of "corner case",
> meant mainly for testing/development, not production. Is there such a big
> advantage in using oVirt for a single host, compared to virt-manager?

The main advantages of ovirt over virt-manager is the access-control and
remote-access capabilities.  Specifically, I have several users which have
different access to different VMs and their consoles.  Without providing
ssh access to the host, I wasn't sure how 

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Yedidyah Bar David]

On Sun, Dec 6, 2020 at 8:14 PM Derek Atkins  wrote:
>
> Hi again,
>
> I also noticed that ca.pem was not updated -- it's still using Sha1.

You are right - we didn't make engine-setup recreate existing certs
for this - "Renew" deals with other stuff [1]. We only change the
default for new ones [2], and wrote a procedure [3][4] for doing this
manually. At the time, this wasn't mandatory - browsers didn't reject
sha1. Perhaps now it should be.

[1] 
https://www.ovirt.org/develop/release-management/features/infra/pki-renew.html

[2] https://gerrit.ovirt.org/c/ovirt-engine/+/65927 (I see that I even
didn't open a bug for this at the time, not sure why)

[3] 
https://www.ovirt.org/documentation/upgrade_guide/#Replacing_SHA-1_Certificates_with_SHA-256_Certificates_4-2_local_db

[4] https://bugzilla.redhat.com/show_bug.cgi?id=1420577 (This is a RHV
doc bug, and the result was an update to RHV docs, which eventually
were also published as [3] above)

> I don't know if this will be an issue with remote-viewer if I wind up
> refreshing the host cert?

As I said, at the time it didn't seem to be mandatory, and docs seemed
to be enough. If you feel otherwise, please open a bug.

I think there is a difference, or at least there was, between what browsers
did/do with https certs, and what they did with CA certs.

If you had a CA cert already accepted/imported/trusted by the browser,
and then you entered a site with a cert signed by this CA, but with a
SHA1 signature, this was one separate case. Browsers started warning/
rejecting them earlier.

If you have a CA cert with a SHA1 signature, and want to import that to
a browser, that's another case. I didn't test recently (or much over time,
other than working on these bugs) with recent browsers, but I think it
took longer until they rejected (if indeed they do - not sure all of them
do).

Best regards,

>
> -derek
>
> On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
> > On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
> >>
> >> Hi,
> >>
> >> I've got a single-host hosted-engine deployment that I originally
> >> installed with 4.0 and have upgraded over the years to 4.3.10.  I and
> >> some
> >> of my users have upgraded remote-viewer and now I get an error when I
> >> try
> >> to view the console of my VMs:
> >>
> >> (remote-viewer:8252): Spice-WARNING **: 11:30:41.806:
> >> ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify:
> >> Error
> >> in server certificate verification: CA signature digest algorithm too
> >> weak
> >> (num=68:depth0:/O=/CN=)
> >>
> >> I am 99.99% sure this is because the old certs use SHA1.
> >>
> >> I reran engine-setup on the engine and it asked me if I wanted to renew
> >> the PKI, and I answered yes.  This replaced many[1] of the certificates
> >> in
> >> /etc/pki/ovirt-engine/certs on the engine, but it did not update the
> >> Host's  certificate.
> >
> > Indeed.
> >
> >>
> >> All the documentation I've seen says that to refresh this certificate I
> >> need to put the host into maintenance mode and then re-enroll..  However
> >> I
> >> cannot do that, because this is a single-host system so I cannot put the
> >> host in local mode -- there is no place to migrate the VMs (let alone
> >> the
> >> Engine VM).
> >>
> >> So  Is there a command-line way to re-enroll manually and update the
> >> host certs?
> >
> > I don't think you'll find anything like this.
> >
> > People did come up in the past with various procedure to hack pki like
> > what
> > you want, but these are, generally speaking, quite fragile - usually do
> > not
> > get updated over versions etc.
> >
> > I am pretty certain the only way to do this using "official" tools/docs
> > is:
> >
> > 1. Stop all VMs except for the engine one.
> >
> > 2. Take a backup with engine-backup.
> >
> > 3. Stop the engine VM.
> >
> > 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
> >
> > 5. Provision the host again as a hosted-engine host, using
> > '--restore-from-file'.
> > Either using new storage for the engine, or after cleaning up the existing
> > hosted-engine storage.
> >
> > If you still want to try doing this manually, then the tool to use is
> > pki-enroll-request.sh. IIRC it's documented. You should find what
> > keys/certs
> > you want to replace, generate new keys and CSRs (or use existing keys and
> > generate CSRs, or even use existing CSRs if you find them), copy to the
> > engine,
> > sign with pki-enroll-request.sh, then copy the generated cert to the host.
> > I am
> > almost certain there is no way to tell vdsm (and other processes) to
> > reload
> > the certs, so you'll have to restart it (them) - and this usually
> > requires putting
> > the host in maintenance (and therefore stop (migrate) all VMs).
> >
> >>  Or some other way to get all the leftover certs renewed?
> >
> > Which ones, specifically?
> >
> >>
> >> Thanks,
> >>
> >> -derek
> >>
> >> [1] Not only did it not update the Host's cert, it did not update any 

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Yedidyah Bar David]

On Sun, Dec 6, 2020 at 7:49 PM Derek Atkins  wrote:
>
> Hi Didi,
>
> One more question:
>
> Can you verify that etc/pki/libvirt/clientcert.pem,
> etc/pki/vdsm/certs/vdsmcert.pem, and
> etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same
> certificate (on the host)?  By a quick find | grep all three of these
> files appear to be the .cer certificate file?

Yes, and also vdsm/libvirt-vnc/server-cert.pem .
-- 
Didi
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SE3OYSJFQJAIQJRZ54E4Z4DCFZ6HZWEV/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Yedidyah Bar David]

On Sun, Dec 6, 2020 at 7:25 PM Derek Atkins  wrote:
>
> HI,
>
> On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
> > On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
> [snip]
> >> So  Is there a command-line way to re-enroll manually and update the
> >> host certs?
> >
> > I don't think you'll find anything like this.
> >
> > People did come up in the past with various procedure to hack pki like
> > what
> > you want, but these are, generally speaking, quite fragile - usually do
> > not
> > get updated over versions etc.
> >
> > I am pretty certain the only way to do this using "official" tools/docs
> > is:
> >
> > 1. Stop all VMs except for the engine one.
> >
> > 2. Take a backup with engine-backup.
> >
> > 3. Stop the engine VM.
> >
> > 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
> >
> > 5. Provision the host again as a hosted-engine host, using
> > '--restore-from-file'.
> > Either using new storage for the engine, or after cleaning up the existing
> > hosted-engine storage.
>
> If I were to go this route I might as well upgrade to EL8 / 4.4 at the
> same time.  However, I would rather not do that; I consider that a very
> dangerous operation, with a generally too-high probability of failure.

I assume you'll do that anyway, some time later, no? So why not now?
I think 4.4.3 is a pretty good version.

I agree it's "dangerous" in the sense that it involves lots of rather
complex actions, some of which are hard to debug/fix if they fail.

But: You can plan and test ahead on a test machine somewhere - even
a VM, with nested-kvm. Just make sure it has no access to your host
and storage (network-wise), so that it does not try to manage/access
them.

>
> > If you still want to try doing this manually, then the tool to use is
> > pki-enroll-request.sh. IIRC it's documented. You should find what
> > keys/certs
> > you want to replace, generate new keys and CSRs (or use existing keys and
> > generate CSRs, or even use existing CSRs if you find them), copy to the
> > engine,
> > sign with pki-enroll-request.sh, then copy the generated cert to the host.
>
> Thanks.  I will look into this method.
>
> > I am
> > almost certain there is no way to tell vdsm (and other processes) to
> > reload
> > the certs, so you'll have to restart it (them) - and this usually
> > requires putting
> > the host in maintenance (and therefore stop (migrate) all VMs).
>
> I don't mind stopping the VMs in order to reboot the host if I can plan
> that.  My understanding is that because there is no place to migrate the
> hosted-engine, that implies even I stop all the other VMs, I still cannot
> put the host into maintenance mode.  Is my understanding correct?

You are right - for a single host, you should do something like this:

1. Stop all VMs except for the engine

2. Put it to global maintenance

3. Stop the engine VM ('poweroff' from inside it, or 'hosted-engine
--vm-shutdown')

4. Restart whatever services you want, or just reboot

5. Exit global maintenance

6. Engine VM should be started automatically in a few minutes. If it
does not, you
can 'hosted-engine --vm-start'. You can monitor agent.log in the meanwhile.

>
> >>  Or some other way to get all the leftover certs renewed?
> >
> > Which ones, specifically?
>
> I think I listed them all:  *.cer and vmconsole*.cer on the engine,
> and of course everything on the host itself.

engine-setup is not designed to touch hosts.

Hosts should be managed by the engine, usually.

>
> Does it matter that ca.der didn't change?  I don't know if that is a
> self-signed cert that might be problematic?

ca.der is not used by anything, you can ignore it. The private key of
the CA is in /etc/pki/ovirt-engine/private/ca.pem, and the public key
is in /etc/pki/ovirt-engine/ca.pem. That's what all tools use.

>
> >>
> >> Thanks,
> >>
> >> -derek
> >>
> >> [1] Not only did it not update the Host's cert, it did not update any of
> >> the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/,
> >> and
> >> obviously nothing in /etc/pki/ on the host itself.
> >
> > AFAIR no process uses these certs as such. There are only processes that
> > use
> > the ssh-format keys extracted from them, which do not include a signature
> > (sha1 or whatever).
> >
> > If you think I am wrong, and/or notice other certs that need to be
> > regenerated,
> > that's a bug - please open one. Thanks!
>
> I have not noticed anything, yet, but I have not restarted the host or
> vdsm since I re-ran engine-setup.
>
> > Re remote-viewer/spice: You didn't say if you tried again after
> > engine-setup
> > and what happened. In any case, this is unrelated to vmconsole (which is
> > for
> > serial consoles, using ssh). But you might still need to regenerate the
> > host
> > cert.
>
> Sorry, I thought I did.  Yes, I did try re-running remote-viewer after
> running engine-setup.  There was no change in the console.vv file (except
> of course for the password and sso-token), so 

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Derek Atkins]

Hi again,

I also noticed that ca.pem was not updated -- it's still using Sha1.
I don't know if this will be an issue with remote-viewer if I wind up
refreshing the host cert?

-derek

On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
> On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
>>
>> Hi,
>>
>> I've got a single-host hosted-engine deployment that I originally
>> installed with 4.0 and have upgraded over the years to 4.3.10.  I and
>> some
>> of my users have upgraded remote-viewer and now I get an error when I
>> try
>> to view the console of my VMs:
>>
>> (remote-viewer:8252): Spice-WARNING **: 11:30:41.806:
>> ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify:
>> Error
>> in server certificate verification: CA signature digest algorithm too
>> weak
>> (num=68:depth0:/O=/CN=)
>>
>> I am 99.99% sure this is because the old certs use SHA1.
>>
>> I reran engine-setup on the engine and it asked me if I wanted to renew
>> the PKI, and I answered yes.  This replaced many[1] of the certificates
>> in
>> /etc/pki/ovirt-engine/certs on the engine, but it did not update the
>> Host's  certificate.
>
> Indeed.
>
>>
>> All the documentation I've seen says that to refresh this certificate I
>> need to put the host into maintenance mode and then re-enroll..  However
>> I
>> cannot do that, because this is a single-host system so I cannot put the
>> host in local mode -- there is no place to migrate the VMs (let alone
>> the
>> Engine VM).
>>
>> So  Is there a command-line way to re-enroll manually and update the
>> host certs?
>
> I don't think you'll find anything like this.
>
> People did come up in the past with various procedure to hack pki like
> what
> you want, but these are, generally speaking, quite fragile - usually do
> not
> get updated over versions etc.
>
> I am pretty certain the only way to do this using "official" tools/docs
> is:
>
> 1. Stop all VMs except for the engine one.
>
> 2. Take a backup with engine-backup.
>
> 3. Stop the engine VM.
>
> 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
>
> 5. Provision the host again as a hosted-engine host, using
> '--restore-from-file'.
> Either using new storage for the engine, or after cleaning up the existing
> hosted-engine storage.
>
> If you still want to try doing this manually, then the tool to use is
> pki-enroll-request.sh. IIRC it's documented. You should find what
> keys/certs
> you want to replace, generate new keys and CSRs (or use existing keys and
> generate CSRs, or even use existing CSRs if you find them), copy to the
> engine,
> sign with pki-enroll-request.sh, then copy the generated cert to the host.
> I am
> almost certain there is no way to tell vdsm (and other processes) to
> reload
> the certs, so you'll have to restart it (them) - and this usually
> requires putting
> the host in maintenance (and therefore stop (migrate) all VMs).
>
>>  Or some other way to get all the leftover certs renewed?
>
> Which ones, specifically?
>
>>
>> Thanks,
>>
>> -derek
>>
>> [1] Not only did it not update the Host's cert, it did not update any of
>> the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/,
>> and
>> obviously nothing in /etc/pki/ on the host itself.
>
> AFAIR no process uses these certs as such. There are only processes that
> use
> the ssh-format keys extracted from them, which do not include a signature
> (sha1 or whatever).
>
> If you think I am wrong, and/or notice other certs that need to be
> regenerated,
> that's a bug - please open one. Thanks!
>
> Re remote-viewer/spice: You didn't say if you tried again after
> engine-setup
> and what happened. In any case, this is unrelated to vmconsole (which is
> for
> serial consoles, using ssh). But you might still need to regenerate the
> host
> cert.
>
> BTW: You can try using novnc and websocket-proxy - engine-setup does
> update
> the cert for the latter, so this might work as-is.
>
> Best regards,
> --
> Didi
>
>


-- 
   Derek Atkins 617-623-3745
   de...@ihtfp.com www.ihtfp.com
   Computer and Internet Security Consultant
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UY44RUFT5MWMZ57Q4A4JWEOVPRSLBGTG/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Derek Atkins]

Hi Didi,

One more question:

Can you verify that etc/pki/libvirt/clientcert.pem,
etc/pki/vdsm/certs/vdsmcert.pem, and
etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same
certificate (on the host)?  By a quick find | grep all three of these
files appear to be the .cer certificate file?

-derek

On Sun, December 6, 2020 12:25 pm, Derek Atkins wrote:
> HI,
>
> On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
>> On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
> [snip]
>>> So  Is there a command-line way to re-enroll manually and update
>>> the
>>> host certs?
>>
>> I don't think you'll find anything like this.
>>
>> People did come up in the past with various procedure to hack pki like
>> what
>> you want, but these are, generally speaking, quite fragile - usually do
>> not
>> get updated over versions etc.
>>
>> I am pretty certain the only way to do this using "official" tools/docs
>> is:
>>
>> 1. Stop all VMs except for the engine one.
>>
>> 2. Take a backup with engine-backup.
>>
>> 3. Stop the engine VM.
>>
>> 4. Reinstall the host OS from scratch or use
>> ovirt-hosted-engine-cleanup.
>>
>> 5. Provision the host again as a hosted-engine host, using
>> '--restore-from-file'.
>> Either using new storage for the engine, or after cleaning up the
>> existing
>> hosted-engine storage.
>
> If I were to go this route I might as well upgrade to EL8 / 4.4 at the
> same time.  However, I would rather not do that; I consider that a very
> dangerous operation, with a generally too-high probability of failure.
>
>> If you still want to try doing this manually, then the tool to use is
>> pki-enroll-request.sh. IIRC it's documented. You should find what
>> keys/certs
>> you want to replace, generate new keys and CSRs (or use existing keys
>> and
>> generate CSRs, or even use existing CSRs if you find them), copy to the
>> engine,
>> sign with pki-enroll-request.sh, then copy the generated cert to the
>> host.
>
> Thanks.  I will look into this method.
>
>> I am
>> almost certain there is no way to tell vdsm (and other processes) to
>> reload
>> the certs, so you'll have to restart it (them) - and this usually
>> requires putting
>> the host in maintenance (and therefore stop (migrate) all VMs).
>
> I don't mind stopping the VMs in order to reboot the host if I can plan
> that.  My understanding is that because there is no place to migrate the
> hosted-engine, that implies even I stop all the other VMs, I still cannot
> put the host into maintenance mode.  Is my understanding correct?
>
>>>  Or some other way to get all the leftover certs renewed?
>>
>> Which ones, specifically?
>
> I think I listed them all:  *.cer and vmconsole*.cer on the engine,
> and of course everything on the host itself.
>
> Does it matter that ca.der didn't change?  I don't know if that is a
> self-signed cert that might be problematic?
>
>>>
>>> Thanks,
>>>
>>> -derek
>>>
>>> [1] Not only did it not update the Host's cert, it did not update any
>>> of
>>> the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/,
>>> and
>>> obviously nothing in /etc/pki/ on the host itself.
>>
>> AFAIR no process uses these certs as such. There are only processes that
>> use
>> the ssh-format keys extracted from them, which do not include a
>> signature
>> (sha1 or whatever).
>>
>> If you think I am wrong, and/or notice other certs that need to be
>> regenerated,
>> that's a bug - please open one. Thanks!
>
> I have not noticed anything, yet, but I have not restarted the host or
> vdsm since I re-ran engine-setup.
>
>> Re remote-viewer/spice: You didn't say if you tried again after
>> engine-setup
>> and what happened. In any case, this is unrelated to vmconsole (which is
>> for
>> serial consoles, using ssh). But you might still need to regenerate the
>> host
>> cert.
>
> Sorry, I thought I did.  Yes, I did try re-running remote-viewer after
> running engine-setup.  There was no change in the console.vv file (except
> of course for the password and sso-token), so yes, it failed in the same
> way.
>
> Note, however, that I did not restart vdsm or the host after running
> engine-setup.
>
>> BTW: You can try using novnc and websocket-proxy - engine-setup does
>> update
>> the cert for the latter, so this might work as-is.
>
> Yes, that does work indeed, so as a short-term solution that can work for
> me.  I'll ask my colleague on a Mac if that works for him.
>
> But it would be nice to get remote-viewer working, IMHO, which would
> require a way to renew / refresh the host cert -- which of course would be
> nice to do without having to re-install!
>
> Thanks!!!
>
>> Best regards,
>> --
>> Didi
>
> -derek
>
> --
>Derek Atkins 617-623-3745
>de...@ihtfp.com www.ihtfp.com
>Computer and Internet Security Consultant
>
>


-- 
   Derek Atkins 617-623-3745
   de...@ihtfp.com www.ihtfp.com
   Computer and Internet Security 

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Derek Atkins]

HI,

On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
> On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
[snip]
>> So  Is there a command-line way to re-enroll manually and update the
>> host certs?
>
> I don't think you'll find anything like this.
>
> People did come up in the past with various procedure to hack pki like
> what
> you want, but these are, generally speaking, quite fragile - usually do
> not
> get updated over versions etc.
>
> I am pretty certain the only way to do this using "official" tools/docs
> is:
>
> 1. Stop all VMs except for the engine one.
>
> 2. Take a backup with engine-backup.
>
> 3. Stop the engine VM.
>
> 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
>
> 5. Provision the host again as a hosted-engine host, using
> '--restore-from-file'.
> Either using new storage for the engine, or after cleaning up the existing
> hosted-engine storage.

If I were to go this route I might as well upgrade to EL8 / 4.4 at the
same time.  However, I would rather not do that; I consider that a very
dangerous operation, with a generally too-high probability of failure.

> If you still want to try doing this manually, then the tool to use is
> pki-enroll-request.sh. IIRC it's documented. You should find what
> keys/certs
> you want to replace, generate new keys and CSRs (or use existing keys and
> generate CSRs, or even use existing CSRs if you find them), copy to the
> engine,
> sign with pki-enroll-request.sh, then copy the generated cert to the host.

Thanks.  I will look into this method.

> I am
> almost certain there is no way to tell vdsm (and other processes) to
> reload
> the certs, so you'll have to restart it (them) - and this usually
> requires putting
> the host in maintenance (and therefore stop (migrate) all VMs).

I don't mind stopping the VMs in order to reboot the host if I can plan
that.  My understanding is that because there is no place to migrate the
hosted-engine, that implies even I stop all the other VMs, I still cannot
put the host into maintenance mode.  Is my understanding correct?

>>  Or some other way to get all the leftover certs renewed?
>
> Which ones, specifically?

I think I listed them all:  *.cer and vmconsole*.cer on the engine,
and of course everything on the host itself.

Does it matter that ca.der didn't change?  I don't know if that is a
self-signed cert that might be problematic?

>>
>> Thanks,
>>
>> -derek
>>
>> [1] Not only did it not update the Host's cert, it did not update any of
>> the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/,
>> and
>> obviously nothing in /etc/pki/ on the host itself.
>
> AFAIR no process uses these certs as such. There are only processes that
> use
> the ssh-format keys extracted from them, which do not include a signature
> (sha1 or whatever).
>
> If you think I am wrong, and/or notice other certs that need to be
> regenerated,
> that's a bug - please open one. Thanks!

I have not noticed anything, yet, but I have not restarted the host or
vdsm since I re-ran engine-setup.

> Re remote-viewer/spice: You didn't say if you tried again after
> engine-setup
> and what happened. In any case, this is unrelated to vmconsole (which is
> for
> serial consoles, using ssh). But you might still need to regenerate the
> host
> cert.

Sorry, I thought I did.  Yes, I did try re-running remote-viewer after
running engine-setup.  There was no change in the console.vv file (except
of course for the password and sso-token), so yes, it failed in the same
way.

Note, however, that I did not restart vdsm or the host after running
engine-setup.

> BTW: You can try using novnc and websocket-proxy - engine-setup does
> update
> the cert for the latter, so this might work as-is.

Yes, that does work indeed, so as a short-term solution that can work for
me.  I'll ask my colleague on a Mac if that works for him.

But it would be nice to get remote-viewer working, IMHO, which would
require a way to renew / refresh the host cert -- which of course would be
nice to do without having to re-install!

Thanks!!!

> Best regards,
> --
> Didi

-derek

-- 
   Derek Atkins 617-623-3745
   de...@ihtfp.com www.ihtfp.com
   Computer and Internet Security Consultant
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4AGI6SIPIP6JRU4SYLTXL5YGP5VPL462/

[ovirt-users] Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?

[Yedidyah Bar David]

On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins  wrote:
>
> Hi,
>
> I've got a single-host hosted-engine deployment that I originally
> installed with 4.0 and have upgraded over the years to 4.3.10.  I and some
> of my users have upgraded remote-viewer and now I get an error when I try
> to view the console of my VMs:
>
> (remote-viewer:8252): Spice-WARNING **: 11:30:41.806:
> ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error
> in server certificate verification: CA signature digest algorithm too weak
> (num=68:depth0:/O=/CN=)
>
> I am 99.99% sure this is because the old certs use SHA1.
>
> I reran engine-setup on the engine and it asked me if I wanted to renew
> the PKI, and I answered yes.  This replaced many[1] of the certificates in
> /etc/pki/ovirt-engine/certs on the engine, but it did not update the
> Host's  certificate.

Indeed.

>
> All the documentation I've seen says that to refresh this certificate I
> need to put the host into maintenance mode and then re-enroll..  However I
> cannot do that, because this is a single-host system so I cannot put the
> host in local mode -- there is no place to migrate the VMs (let alone the
> Engine VM).
>
> So  Is there a command-line way to re-enroll manually and update the
> host certs?

I don't think you'll find anything like this.

People did come up in the past with various procedure to hack pki like what
you want, but these are, generally speaking, quite fragile - usually do not
get updated over versions etc.

I am pretty certain the only way to do this using "official" tools/docs is:

1. Stop all VMs except for the engine one.

2. Take a backup with engine-backup.

3. Stop the engine VM.

4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.

5. Provision the host again as a hosted-engine host, using
'--restore-from-file'.
Either using new storage for the engine, or after cleaning up the existing
hosted-engine storage.

If you still want to try doing this manually, then the tool to use is
pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs
you want to replace, generate new keys and CSRs (or use existing keys and
generate CSRs, or even use existing CSRs if you find them), copy to the engine,
sign with pki-enroll-request.sh, then copy the generated cert to the host. I am
almost certain there is no way to tell vdsm (and other processes) to reload
the certs, so you'll have to restart it (them) - and this usually
requires putting
the host in maintenance (and therefore stop (migrate) all VMs).

>  Or some other way to get all the leftover certs renewed?

Which ones, specifically?

>
> Thanks,
>
> -derek
>
> [1] Not only did it not update the Host's cert, it did not update any of
> the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and
> obviously nothing in /etc/pki/ on the host itself.

AFAIR no process uses these certs as such. There are only processes that use
the ssh-format keys extracted from them, which do not include a signature
(sha1 or whatever).

If you think I am wrong, and/or notice other certs that need to be regenerated,
that's a bug - please open one. Thanks!

Re remote-viewer/spice: You didn't say if you tried again after engine-setup
and what happened. In any case, this is unrelated to vmconsole (which is for
serial consoles, using ssh). But you might still need to regenerate the host
cert.

BTW: You can try using novnc and websocket-proxy - engine-setup does update
the cert for the latter, so this might work as-is.

Best regards,
-- 
Didi
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FEJUG6F74JPFAA63MEA4GWPBCUZRCF66/