[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-27 Thread Yedidyah Bar David 
On Tue, Dec 27, 2022 at 6:18 PM Gilboa Davara  wrote:
>
> Hello,
>
> On Tue, Dec 27, 2022 at 8:40 AM Yedidyah Bar David  wrote:
>>
>> > Add issue https://github.com/oVirt/ovirt-engine/issues/784
>>
>> Sorry, I do not follow. Is your immediate obstacle being that
>> engine-setup refuses to continue, saying "Hosted Engine HA is in
>> Global Maintenance mode."?
>>
>> You can cause it to ignore this test by passing
>> 'OVESETUP_CONFIG/continueSetupOnHEVM=bool:True' (in the answer file or
>> --otopi-environment).
>>
>> We recently added an option 'engine-setup
>> --show-environment-documentation', exactly for this env key, see also:
>>
>> https://bugzilla.redhat.com/show_bug.ccontinueSetupOnHEVM=bool:Truegi?id=1700460
>>
>> Best regards,
>> --
>> Didi
>>
>
> I actually managed to bypass the check by editing he.py and deleting the 
> "raise" statement, preventing hosted-engine from bombing out because it 
> wasn't able to connect to the nodes.
> From there I managed to renew the certificates (see second mail), and even 
> connected two of the 3 nodes successfully (I had to create new temporary vdsm 
> certificates, get them semi-connected to the engine, and then "re-enroll 
> certificates" from the UI. Once I had a limping cluster up, I shut everything 
> down cleanly, and... and redeployed the cluster from scratch. (with all the 
> failed attempts, my HE was completely busted).
> That said, I wonder if having to short circuit the environment variable isn't 
> a bit over-complicated, given the considerable number of cert related issues.

I do not think it's "over complicated" in any technical sense - just
one command line to copy/paste from somewhere. I'd say it's mainly
that knowing that this is the solution to your exact problem is the
hard thing.

>
> But thanks for the heads-up.
>
> Q: I'm willing to try and document all the steps I did, in my semi-success 
> attempt to save my cluster.

I think that would be great.

> That said, I rather not document wrong / broken steps. Can anyone @RH review 
> my writeup?

Sure! But consider how you intend to publish it. If as something like
a blog post (on ovirt.org or your own blog or whatever), that's less
"authoritative" and understandably more local/specific. If you
consider integrating it into the official guides, that's more
delicate.
-- 
Didi
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZS3GZUZZDFUUX4RR3KQT4BO6TQOXHYF4/

[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-27 Thread Gilboa Davara 
Hello,

On Tue, Dec 27, 2022 at 8:40 AM Yedidyah Bar David  wrote:

> > Add issue https://github.com/oVirt/ovirt-engine/issues/784
>
> Sorry, I do not follow. Is your immediate obstacle being that
> engine-setup refuses to continue, saying "Hosted Engine HA is in
> Global Maintenance mode."?
>
> You can cause it to ignore this test by passing
> 'OVESETUP_CONFIG/continueSetupOnHEVM=bool:True' (in the answer file or
> --otopi-environment).
>
> We recently added an option 'engine-setup
> --show-environment-documentation', exactly for this env key, see also:
>
>
> https://bugzilla.redhat.com/show_bug.ccontinueSetupOnHEVM=bool:Truegi?id=1700460
> 
>
> Best regards,
> --
> Didi
>
>
I actually managed to bypass the check by editing he.py and deleting the
"raise" statement, preventing hosted-engine from bombing out because it
wasn't able to connect to the nodes.
>From there I managed to renew the certificates (see second mail), and even
connected two of the 3 nodes successfully (I had to create new temporary
vdsm certificates, get them semi-connected to the engine, and then
"re-enroll certificates" from the UI. Once I had a limping cluster up, I
shut everything down cleanly, and... and redeployed the cluster from
scratch. (with all the failed attempts, my HE was completely busted).
That said, I wonder if having to short circuit the environment variable
isn't a bit over-complicated, given the considerable number of cert related
issues.

But thanks for the heads-up.

Q: I'm willing to try and document all the steps I did, in my semi-success
attempt to save my cluster.
That said, I rather not document wrong / broken steps. Can anyone @RH
review my writeup?

- Gilboa
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WAX664FLYQMMV6AG54YVMIIRGIHYJAOT/

[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-26 Thread Yedidyah Bar David 
On Tue, Dec 27, 2022 at 8:39 AM Yedidyah Bar David  wrote:
>
> On Sun, Dec 25, 2022 at 5:15 PM Gilboa Davara  wrote:
> >
> >
> >
> > On Sun, Dec 25, 2022 at 12:37 PM Gilboa Davara  wrote:
> >>
> >> On Sun, Dec 25, 2022 at 12:36 PM Gilboa Davara  wrote:
> >>>
> >>> Hello all,
> >>>
> >>> Even though I do my best to keep track of the certificate issue date 
> >>> across my different clusters, I somehow missed the vdsm certificate 
> >>> expiration in one of my clusters.
> >>> Now I have an active cluster with multiple nodes (self-hosted / gluster 
> >>> storage), vdsm service is down on all nodes (due to certificate 
> >>> expiration) - hence, I cannot get the cluster into global maintenance 
> >>> mode (vdsms are down), and I cannot access my engine (to renew the engine 
> >>> certificates / re-enroll hosts).
> >>> How can manual renew the host certificate?
> >>>
> >>> Thanks,
> >>> Gilboa
> >>
> >>
> >> P.S. CentOS 8 Streams engine and host, ovirt v4.5.3 (I think).
> >>
> >> - Gilboa
> >
> >
> > Managed to find an old email in this group (that I saved...)
> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/56QU2AD7YUX2VZUP4NZMRFXK32MJM7QE/
> >
> > This got the nodes working... but the engine (GRRR) still cannot connect to 
> > the nodes (I assume it has expired certs as well), hence, it cannot detect 
> > the cluster is in global maintenance mode, and cannot run engine-setup.
> >
> > Add issue https://github.com/oVirt/ovirt-engine/issues/784
>
> Sorry, I do not follow. Is your immediate obstacle being that
> engine-setup refuses to continue, saying "Hosted Engine HA is in
> Global Maintenance mode."?
>
> You can cause it to ignore this test by passing
> 'OVESETUP_CONFIG/continueSetupOnHEVM=bool:True' (in the answer file or
> --otopi-environment).
>
> We recently added an option 'engine-setup
> --show-environment-documentation', exactly for this env key, see also:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1700460

(BTW, I now see that I warned there against trying to parse the
output, as it might change in the future - and that I indeed actually
already "broke" it, https://github.com/oVirt/otopi/pull/22 . If anyone
volunteers to enhance this - either add some override to otopi calling
textwrap.wrap or perhaps some '--json' option or whatever, great!).
-- 
Didi
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/F6YJ5C23EAEFQSZ76DFCYOCFYLIWRMZT/

[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-26 Thread Yedidyah Bar David 
On Sun, Dec 25, 2022 at 5:15 PM Gilboa Davara  wrote:
>
>
>
> On Sun, Dec 25, 2022 at 12:37 PM Gilboa Davara  wrote:
>>
>> On Sun, Dec 25, 2022 at 12:36 PM Gilboa Davara  wrote:
>>>
>>> Hello all,
>>>
>>> Even though I do my best to keep track of the certificate issue date across 
>>> my different clusters, I somehow missed the vdsm certificate expiration in 
>>> one of my clusters.
>>> Now I have an active cluster with multiple nodes (self-hosted / gluster 
>>> storage), vdsm service is down on all nodes (due to certificate expiration) 
>>> - hence, I cannot get the cluster into global maintenance mode (vdsms are 
>>> down), and I cannot access my engine (to renew the engine certificates / 
>>> re-enroll hosts).
>>> How can manual renew the host certificate?
>>>
>>> Thanks,
>>> Gilboa
>>
>>
>> P.S. CentOS 8 Streams engine and host, ovirt v4.5.3 (I think).
>>
>> - Gilboa
>
>
> Managed to find an old email in this group (that I saved...)
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/56QU2AD7YUX2VZUP4NZMRFXK32MJM7QE/
>
> This got the nodes working... but the engine (GRRR) still cannot connect to 
> the nodes (I assume it has expired certs as well), hence, it cannot detect 
> the cluster is in global maintenance mode, and cannot run engine-setup.
>
> Add issue https://github.com/oVirt/ovirt-engine/issues/784

Sorry, I do not follow. Is your immediate obstacle being that
engine-setup refuses to continue, saying "Hosted Engine HA is in
Global Maintenance mode."?

You can cause it to ignore this test by passing
'OVESETUP_CONFIG/continueSetupOnHEVM=bool:True' (in the answer file or
--otopi-environment).

We recently added an option 'engine-setup
--show-environment-documentation', exactly for this env key, see also:

https://bugzilla.redhat.com/show_bug.cgi?id=1700460

Best regards,
-- 
Didi
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L4KZTEI424WD7YU4W2WIT4LELEU2FPAZ/

[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-25 Thread Gilboa Davara 
On Sun, Dec 25, 2022 at 12:37 PM Gilboa Davara  wrote:

> On Sun, Dec 25, 2022 at 12:36 PM Gilboa Davara  wrote:
>
>> Hello all,
>>
>> Even though I do my best to keep track of the certificate issue date
>> across my different clusters, I somehow missed the vdsm certificate
>> expiration in one of my clusters.
>> Now I have an active cluster with multiple nodes (self-hosted / gluster
>> storage), vdsm service is down on all nodes (due to certificate expiration)
>> - hence, I cannot get the cluster into global maintenance mode (vdsms are
>> down), and I cannot access my engine (to renew the engine certificates /
>> re-enroll hosts).
>> How can manual renew the host certificate?
>>
>> Thanks,
>> Gilboa
>>
>
> P.S. CentOS 8 Streams engine and host, ovirt v4.5.3 (I think).
>
> - Gilboa
>

Managed to find an old email in this group (that I saved...)
https://lists.ovirt.org/archives/list/users@ovirt.org/message/56QU2AD7YUX2VZUP4NZMRFXK32MJM7QE/

This got the nodes working... but the engine (GRRR) still cannot connect to
the nodes (I assume it has expired certs as well), hence, it cannot detect
the cluster is in global maintenance mode, and cannot run engine-setup.

Add issue https://github.com/oVirt/ovirt-engine/issues/784

- Gilboa
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TNKL44CGJXEWPRIY6AQNTULH2RM7O357/

[ovirt-users] Re: Certificate expiration w/o warning on all clients. Cluster in zombie state

2022-12-25 Thread Gilboa Davara 
On Sun, Dec 25, 2022 at 12:36 PM Gilboa Davara  wrote:

> Hello all,
>
> Even though I do my best to keep track of the certificate issue date
> across my different clusters, I somehow missed the vdsm certificate
> expiration in one of my clusters.
> Now I have an active cluster with multiple nodes (self-hosted / gluster
> storage), vdsm service is down on all nodes (due to certificate expiration)
> - hence, I cannot get the cluster into global maintenance mode (vdsms are
> down), and I cannot access my engine (to renew the engine certificates /
> re-enroll hosts).
> How can manual renew the host certificate?
>
> Thanks,
> Gilboa
>

P.S. CentOS 8 Streams engine and host, ovirt v4.5.3 (I think).

- Gilboa
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HR3KH4CZPY7COCFGXFSCMAGPZIGCAPQD/