[ovirt-users] Re: Upgraded host, engine now won't boot

[Darrell Budic]

Glad you got into it and got it working. Not sure why it keeps unpausing, could 
open a bug if you wanted.

Yep, the engine-setup is on the engine vm itself, not the hosts. You want 
https://www.ovirt.org/documentation/self-hosted/chap-Maintenance_and_Upgrading_Resources/
 
,
 although if you aren’t running the appliance in the first place, I don’t think 
it all applies. Just log in and run yum update or yum update ovirt* and then 
engine-setup again.


> From: Jim Kusznir 
> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
> Date: September 3, 2018 at 10:21:01 PM CDT
> To: Darrell Budic
> Cc: users
> 
> Ok, finally got it...Had to get a terminal ready with the virsh command and 
> guess what the instance number was, and then run suspend right after starting 
> with --vm-start-paused.  Got it to really be paused, got into the console, 
> booted the old kernel, and have now been repairing a bad yum transactionI 
> *think* I've finished that.
> 
> So, if I understand correctly, after the yum update, I should run 
> engine-setup?  Do I run that inside the engine vm, or on the host its running 
> on?
> 
> BTW: I did look up upgrade procedures on the documentation for the release.  
> It links to two or three levels of other documents, then ends in an error 404.
> 
> --Jim
> 
> On Mon, Sep 3, 2018 at 6:39 PM, Jim Kusznir  > wrote:
> global maintence mode is already on.  hosted-engine --vm-start-paused results 
> in a non-paused VM being started.  Of course, this is executed after 
> hosted-engine --vm-poweroff and suitable time left to let things shut down.
> 
> I just ran another test, and did in fact see the engine was briefly paused, 
> but then was quickly put in the running state.  I don't know by what, though. 
>  Global maintence mode is definitely enabled, every run of the hosted-engine 
> command reminds me!
> 
> 
> 
> 
> 
> On Mon, Sep 3, 2018 at 11:12 AM, Darrell Budic  > wrote:
> Don’t know if there’s anything special, it’s been a while since I’ve needed 
> to start it in paused mode. Try putting it in HA maintenance mode from the 
> CLI and then start it in paused mode maybe?
> 
>> From: Jim Kusznir mailto:j...@palousetech.com>>
>> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
>> Date: September 3, 2018 at 1:08:27 PM CDT
>> 
>> To: Darrell Budic
>> Cc: users
>> 
>> Unfortunately, I seem unable to get connected to the console early enough to 
>> actually see a kernel list.
>> 
>> I've tried the hosted-engine --start-vm-paused command, but it just starts 
>> it (running mode, not paused).  By the time I can get vnc connected, I have 
>> just that last line.  ctrl-alt-del doesn't do anything with it, either.  
>> sending a reset through virsh seems to just kill the VM (it doesn't respawn).
>> 
>> ha seems to have some trouble with this too...Originally I allowed ha to 
>> start it, and it would take it a good long while before it gave up on the 
>> engine and reset it.  It instantly booted to the same crashed state, and 
>> again waited a "good long while" (sorry, never timed it, but I know it was 
>> >5 min).
>> 
>> My current thought is that I need to get the engine started in paused mode, 
>> connect vnc, then unpause it with virsh to catch what is happening.  Is 
>> there any magic to getting it started in paused mode?
>> 
>> On Mon, Sep 3, 2018 at 11:03 AM, Darrell Budic > > wrote:
>> Send it a ctl-alt-delete and see what happens. Possibly try an older kernel 
>> at the grub boot menu. Could also try stopping it with hosted-engine 
>> —vm-stop and let HA reboot it, see if it boots or get onto the console 
>> quickly and try and watch more of the boot.
>> 
>> Ssh and yum upgrade is fine for the OS, although it’s a good idea to enable 
>> Global HA Maintenance first so the HA watchdogs don’t reboot it in the 
>> middle of that. After that, run “engine-setup” again, at least if there are 
>> new ovirt engine updates to be done. Then disable Global HA Maintenance, and 
>> run "shutdown -h now” to stop the Engine VM (rebooting seems to cause it to 
>> exit anyway, HA seems to run it as a single execution VM. Or at least in the 
>> past, it seems to quit anyway on me and shutdown triggered HA faster). Wait 
>> a few minutes, and HA will respawn it on a new instance and you can log into 
>> your engine again.
>>> From: Jim Kusznir mailto:j...@palousetech.com>>
>>> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
>>> Date: September 3, 2018 at 12:45:22 PM CDT
>>> To: Darrell Budic
>>> Cc: users
>>> 
>>> 
>>> Thanks to Jayme who pointed me to the --add-console-password hosted-engine 
>>> command to set a password for vnc.  Using that, I see only the single line:
>>> 
>>> Probing EDD (edd=off to disable)... ok
>>> 
>>> --Jim
>>> 
>>> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kuszni

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

Ok, finally got it...Had to get a terminal ready with the virsh command and
guess what the instance number was, and then run suspend right after
starting with --vm-start-paused.  Got it to really be paused, got into the
console, booted the old kernel, and have now been repairing a bad yum
transactionI *think* I've finished that.

So, if I understand correctly, after the yum update, I should run
engine-setup?  Do I run that inside the engine vm, or on the host its
running on?

BTW: I did look up upgrade procedures on the documentation for the
release.  It links to two or three levels of other documents, then ends in
an error 404.

--Jim

On Mon, Sep 3, 2018 at 6:39 PM, Jim Kusznir  wrote:

> global maintence mode is already on.  hosted-engine --vm-start-paused
> results in a non-paused VM being started.  Of course, this is executed
> after hosted-engine --vm-poweroff and suitable time left to let things shut
> down.
>
> I just ran another test, and did in fact see the engine was briefly
> paused, but then was quickly put in the running state.  I don't know by
> what, though.  Global maintence mode is definitely enabled, every run of
> the hosted-engine command reminds me!
>
>
>
>
>
> On Mon, Sep 3, 2018 at 11:12 AM, Darrell Budic 
> wrote:
>
>> Don’t know if there’s anything special, it’s been a while since I’ve
>> needed to start it in paused mode. Try putting it in HA maintenance mode
>> from the CLI and then start it in paused mode maybe?
>>
>> --
>> *From:* Jim Kusznir 
>> *Subject:* Re: [ovirt-users] Upgraded host, engine now won't boot
>> *Date:* September 3, 2018 at 1:08:27 PM CDT
>>
>> *To:* Darrell Budic
>> *Cc:* users
>>
>> Unfortunately, I seem unable to get connected to the console early enough
>> to actually see a kernel list.
>>
>> I've tried the hosted-engine --start-vm-paused command, but it just
>> starts it (running mode, not paused).  By the time I can get vnc connected,
>> I have just that last line.  ctrl-alt-del doesn't do anything with it,
>> either.  sending a reset through virsh seems to just kill the VM (it
>> doesn't respawn).
>>
>> ha seems to have some trouble with this too...Originally I allowed ha to
>> start it, and it would take it a good long while before it gave up on the
>> engine and reset it.  It instantly booted to the same crashed state, and
>> again waited a "good long while" (sorry, never timed it, but I know it was
>> >5 min).
>>
>> My current thought is that I need to get the engine started in paused
>> mode, connect vnc, then unpause it with virsh to catch what is happening.
>> Is there any magic to getting it started in paused mode?
>>
>> On Mon, Sep 3, 2018 at 11:03 AM, Darrell Budic 
>> wrote:
>>
>>> Send it a ctl-alt-delete and see what happens. Possibly try an older
>>> kernel at the grub boot menu. Could also try stopping it with hosted-engine
>>> —vm-stop and let HA reboot it, see if it boots or get onto the console
>>> quickly and try and watch more of the boot.
>>>
>>> Ssh and yum upgrade is fine for the OS, although it’s a good idea to
>>> enable Global HA Maintenance first so the HA watchdogs don’t reboot it in
>>> the middle of that. After that, run “engine-setup” again, at least if there
>>> are new ovirt engine updates to be done. Then disable Global HA
>>> Maintenance, and run "shutdown -h now” to stop the Engine VM (rebooting
>>> seems to cause it to exit anyway, HA seems to run it as a single execution
>>> VM. Or at least in the past, it seems to quit anyway on me and shutdown
>>> triggered HA faster). Wait a few minutes, and HA will respawn it on a new
>>> instance and you can log into your engine again.
>>>
>>> --
>>> *From:* Jim Kusznir 
>>> *Subject:* Re: [ovirt-users] Upgraded host, engine now won't boot
>>> *Date:* September 3, 2018 at 12:45:22 PM CDT
>>> *To:* Darrell Budic
>>> *Cc:* users
>>>
>>>
>>> Thanks to Jayme who pointed me to the --add-console-password
>>> hosted-engine command to set a password for vnc.  Using that, I see only
>>> the single line:
>>>
>>> Probing EDD (edd=off to disable)... ok
>>>
>>> --Jim
>>>
>>> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir 
>>> wrote:
>>>
 Is there a way to get a graphical console on boot of the engine vm so I
 can see what's causing the failure to boot?

 On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir 
 wrote:

> Thanks; I guess I didn't mention that I started there.
>
> The virsh list shows it in state running, and gluster is showing fully
> online and healed.  However, I cannot bring up a console of the engine VM
> to see why its not booting, even though it shows in running state.
>
> In any case, the hosts and engine were running happily.  I applied the
> latest updates on the host, and the engine went unstable.  I thought, Ok,
> maybe there's an update to ovirt that also needs to be applied to the
> engine, so I ssh'ed in and ran yum update (never did find clear
> 

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

global maintence mode is already on.  hosted-engine --vm-start-paused
results in a non-paused VM being started.  Of course, this is executed
after hosted-engine --vm-poweroff and suitable time left to let things shut
down.

I just ran another test, and did in fact see the engine was briefly paused,
but then was quickly put in the running state.  I don't know by what,
though.  Global maintence mode is definitely enabled, every run of the
hosted-engine command reminds me!





On Mon, Sep 3, 2018 at 11:12 AM, Darrell Budic 
wrote:

> Don’t know if there’s anything special, it’s been a while since I’ve
> needed to start it in paused mode. Try putting it in HA maintenance mode
> from the CLI and then start it in paused mode maybe?
>
> --
> *From:* Jim Kusznir 
> *Subject:* Re: [ovirt-users] Upgraded host, engine now won't boot
> *Date:* September 3, 2018 at 1:08:27 PM CDT
>
> *To:* Darrell Budic
> *Cc:* users
>
> Unfortunately, I seem unable to get connected to the console early enough
> to actually see a kernel list.
>
> I've tried the hosted-engine --start-vm-paused command, but it just starts
> it (running mode, not paused).  By the time I can get vnc connected, I have
> just that last line.  ctrl-alt-del doesn't do anything with it, either.
> sending a reset through virsh seems to just kill the VM (it doesn't
> respawn).
>
> ha seems to have some trouble with this too...Originally I allowed ha to
> start it, and it would take it a good long while before it gave up on the
> engine and reset it.  It instantly booted to the same crashed state, and
> again waited a "good long while" (sorry, never timed it, but I know it was
> >5 min).
>
> My current thought is that I need to get the engine started in paused
> mode, connect vnc, then unpause it with virsh to catch what is happening.
> Is there any magic to getting it started in paused mode?
>
> On Mon, Sep 3, 2018 at 11:03 AM, Darrell Budic 
> wrote:
>
>> Send it a ctl-alt-delete and see what happens. Possibly try an older
>> kernel at the grub boot menu. Could also try stopping it with hosted-engine
>> —vm-stop and let HA reboot it, see if it boots or get onto the console
>> quickly and try and watch more of the boot.
>>
>> Ssh and yum upgrade is fine for the OS, although it’s a good idea to
>> enable Global HA Maintenance first so the HA watchdogs don’t reboot it in
>> the middle of that. After that, run “engine-setup” again, at least if there
>> are new ovirt engine updates to be done. Then disable Global HA
>> Maintenance, and run "shutdown -h now” to stop the Engine VM (rebooting
>> seems to cause it to exit anyway, HA seems to run it as a single execution
>> VM. Or at least in the past, it seems to quit anyway on me and shutdown
>> triggered HA faster). Wait a few minutes, and HA will respawn it on a new
>> instance and you can log into your engine again.
>>
>> --
>> *From:* Jim Kusznir 
>> *Subject:* Re: [ovirt-users] Upgraded host, engine now won't boot
>> *Date:* September 3, 2018 at 12:45:22 PM CDT
>> *To:* Darrell Budic
>> *Cc:* users
>>
>>
>> Thanks to Jayme who pointed me to the --add-console-password
>> hosted-engine command to set a password for vnc.  Using that, I see only
>> the single line:
>>
>> Probing EDD (edd=off to disable)... ok
>>
>> --Jim
>>
>> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir  wrote:
>>
>>> Is there a way to get a graphical console on boot of the engine vm so I
>>> can see what's causing the failure to boot?
>>>
>>> On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir 
>>> wrote:
>>>
 Thanks; I guess I didn't mention that I started there.

 The virsh list shows it in state running, and gluster is showing fully
 online and healed.  However, I cannot bring up a console of the engine VM
 to see why its not booting, even though it shows in running state.

 In any case, the hosts and engine were running happily.  I applied the
 latest updates on the host, and the engine went unstable.  I thought, Ok,
 maybe there's an update to ovirt that also needs to be applied to the
 engine, so I ssh'ed in and ran yum update (never did find clear
 instructions on how one is supposed to maintain the engine, but I did see
 that listed online).  A while later, it reset and never booted again.

 -JIm

 On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic 
 wrote:

> It’s definitely not starting, you’ll have to see if you can figure out
> why. A couple things to try:
>
> - Check "virsh list" and see if it’s running, or paused for storage.
> (google "virsh saslpasswd2
> ”
> if you need to add a user to do this with, it’s per host)
> -  It’s hyper converged, so check your gluster volume for healing
> and/or split brains and wait/resolve those.
> - check “gluster peer status” and on each host and make sure you

[ovirt-users] Re: Upgraded host, engine now won't boot

[Darrell Budic]

Don’t know if there’s anything special, it’s been a while since I’ve needed to 
start it in paused mode. Try putting it in HA maintenance mode from the CLI and 
then start it in paused mode maybe?

> From: Jim Kusznir 
> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
> Date: September 3, 2018 at 1:08:27 PM CDT
> To: Darrell Budic
> Cc: users
> 
> Unfortunately, I seem unable to get connected to the console early enough to 
> actually see a kernel list.
> 
> I've tried the hosted-engine --start-vm-paused command, but it just starts it 
> (running mode, not paused).  By the time I can get vnc connected, I have just 
> that last line.  ctrl-alt-del doesn't do anything with it, either.  sending a 
> reset through virsh seems to just kill the VM (it doesn't respawn).
> 
> ha seems to have some trouble with this too...Originally I allowed ha to 
> start it, and it would take it a good long while before it gave up on the 
> engine and reset it.  It instantly booted to the same crashed state, and 
> again waited a "good long while" (sorry, never timed it, but I know it was >5 
> min).
> 
> My current thought is that I need to get the engine started in paused mode, 
> connect vnc, then unpause it with virsh to catch what is happening.  Is there 
> any magic to getting it started in paused mode?
> 
> On Mon, Sep 3, 2018 at 11:03 AM, Darrell Budic  > wrote:
> Send it a ctl-alt-delete and see what happens. Possibly try an older kernel 
> at the grub boot menu. Could also try stopping it with hosted-engine —vm-stop 
> and let HA reboot it, see if it boots or get onto the console quickly and try 
> and watch more of the boot.
> 
> Ssh and yum upgrade is fine for the OS, although it’s a good idea to enable 
> Global HA Maintenance first so the HA watchdogs don’t reboot it in the middle 
> of that. After that, run “engine-setup” again, at least if there are new 
> ovirt engine updates to be done. Then disable Global HA Maintenance, and run 
> "shutdown -h now” to stop the Engine VM (rebooting seems to cause it to exit 
> anyway, HA seems to run it as a single execution VM. Or at least in the past, 
> it seems to quit anyway on me and shutdown triggered HA faster). Wait a few 
> minutes, and HA will respawn it on a new instance and you can log into your 
> engine again.
>> From: Jim Kusznir mailto:j...@palousetech.com>>
>> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
>> Date: September 3, 2018 at 12:45:22 PM CDT
>> To: Darrell Budic
>> Cc: users
>> 
>> 
>> Thanks to Jayme who pointed me to the --add-console-password hosted-engine 
>> command to set a password for vnc.  Using that, I see only the single line:
>> 
>> Probing EDD (edd=off to disable)... ok
>> 
>> --Jim
>> 
>> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir > > wrote:
>> Is there a way to get a graphical console on boot of the engine vm so I can 
>> see what's causing the failure to boot?
>> 
>> On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir > > wrote:
>> Thanks; I guess I didn't mention that I started there.
>> 
>> The virsh list shows it in state running, and gluster is showing fully 
>> online and healed.  However, I cannot bring up a console of the engine VM to 
>> see why its not booting, even though it shows in running state.
>> 
>> In any case, the hosts and engine were running happily.  I applied the 
>> latest updates on the host, and the engine went unstable.  I thought, Ok, 
>> maybe there's an update to ovirt that also needs to be applied to the 
>> engine, so I ssh'ed in and ran yum update (never did find clear instructions 
>> on how one is supposed to maintain the engine, but I did see that listed 
>> online).  A while later, it reset and never booted again.
>> 
>> -JIm
>> 
>> On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic > > wrote:
>> It’s definitely not starting, you’ll have to see if you can figure out why. 
>> A couple things to try:
>> 
>> - Check "virsh list" and see if it’s running, or paused for storage. (google 
>> "virsh saslpasswd2 
>> ”
>>  if you need to add a user to do this with, it’s per host)
>> -  It’s hyper converged, so check your gluster volume for healing and/or 
>> split brains and wait/resolve those.
>> - check “gluster peer status” and on each host and make sure your gluster 
>> hosts are all talking. I’ve seen an upgrade screwup the firewall, easy fix 
>> is to add a rule to allow the hosts to talk to each other on your gluster 
>> network, no questions asked (-j ACCEPT, no port, etc).
>> 
>> Good luck!
>> 
>>> From: Jim Kusznir mailto:j...@palousetech.com>>
>>> Subject: [ovirt-users] Upgraded host, engine now won't boot
>>> Date: September 1, 2018 at 8:38:12 PM CDT
>>> To: users
>>> 
>>> Hello:
>>> 
>>> I saw that there were updates to my ovirt-4.2 3 node hyperconverged system, 
>>>

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

Unfortunately, I seem unable to get connected to the console early enough
to actually see a kernel list.

I've tried the hosted-engine --start-vm-paused command, but it just starts
it (running mode, not paused).  By the time I can get vnc connected, I have
just that last line.  ctrl-alt-del doesn't do anything with it, either.
sending a reset through virsh seems to just kill the VM (it doesn't
respawn).

ha seems to have some trouble with this too...Originally I allowed ha to
start it, and it would take it a good long while before it gave up on the
engine and reset it.  It instantly booted to the same crashed state, and
again waited a "good long while" (sorry, never timed it, but I know it was
>5 min).

My current thought is that I need to get the engine started in paused mode,
connect vnc, then unpause it with virsh to catch what is happening.  Is
there any magic to getting it started in paused mode?

On Mon, Sep 3, 2018 at 11:03 AM, Darrell Budic 
wrote:

> Send it a ctl-alt-delete and see what happens. Possibly try an older
> kernel at the grub boot menu. Could also try stopping it with hosted-engine
> —vm-stop and let HA reboot it, see if it boots or get onto the console
> quickly and try and watch more of the boot.
>
> Ssh and yum upgrade is fine for the OS, although it’s a good idea to
> enable Global HA Maintenance first so the HA watchdogs don’t reboot it in
> the middle of that. After that, run “engine-setup” again, at least if there
> are new ovirt engine updates to be done. Then disable Global HA
> Maintenance, and run "shutdown -h now” to stop the Engine VM (rebooting
> seems to cause it to exit anyway, HA seems to run it as a single execution
> VM. Or at least in the past, it seems to quit anyway on me and shutdown
> triggered HA faster). Wait a few minutes, and HA will respawn it on a new
> instance and you can log into your engine again.
>
> --
> *From:* Jim Kusznir 
> *Subject:* Re: [ovirt-users] Upgraded host, engine now won't boot
> *Date:* September 3, 2018 at 12:45:22 PM CDT
> *To:* Darrell Budic
> *Cc:* users
>
>
> Thanks to Jayme who pointed me to the --add-console-password hosted-engine
> command to set a password for vnc.  Using that, I see only the single line:
>
> Probing EDD (edd=off to disable)... ok
>
> --Jim
>
> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir  wrote:
>
>> Is there a way to get a graphical console on boot of the engine vm so I
>> can see what's causing the failure to boot?
>>
>> On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir  wrote:
>>
>>> Thanks; I guess I didn't mention that I started there.
>>>
>>> The virsh list shows it in state running, and gluster is showing fully
>>> online and healed.  However, I cannot bring up a console of the engine VM
>>> to see why its not booting, even though it shows in running state.
>>>
>>> In any case, the hosts and engine were running happily.  I applied the
>>> latest updates on the host, and the engine went unstable.  I thought, Ok,
>>> maybe there's an update to ovirt that also needs to be applied to the
>>> engine, so I ssh'ed in and ran yum update (never did find clear
>>> instructions on how one is supposed to maintain the engine, but I did see
>>> that listed online).  A while later, it reset and never booted again.
>>>
>>> -JIm
>>>
>>> On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic 
>>> wrote:
>>>
 It’s definitely not starting, you’ll have to see if you can figure out
 why. A couple things to try:

 - Check "virsh list" and see if it’s running, or paused for storage.
 (google "virsh saslpasswd2
 ”
 if you need to add a user to do this with, it’s per host)
 -  It’s hyper converged, so check your gluster volume for healing
 and/or split brains and wait/resolve those.
 - check “gluster peer status” and on each host and make sure your
 gluster hosts are all talking. I’ve seen an upgrade screwup the firewall,
 easy fix is to add a rule to allow the hosts to talk to each other on your
 gluster network, no questions asked (-j ACCEPT, no port, etc).

 Good luck!

 --
 *From:* Jim Kusznir 
 *Subject:* [ovirt-users] Upgraded host, engine now won't boot
 *Date:* September 1, 2018 at 8:38:12 PM CDT
 *To:* users

 Hello:

 I saw that there were updates to my ovirt-4.2 3 node hyperconverged
 system, so I proceeded to apply them the usual way through the UI.

 At one point, the hosted engine was migrated to one of the upgraded
 hosts, and then went "unstable" on me.  Now, the hosted engine appears to
 be crashed:  It gets powered up, but it never boots up to the point where
 it responds to pings or allows logins.  After a while, the hosted engine
 shows status (via console "hosted-engine --vm-status" command) "Powering
 Down".  It stays there for a long time.

[ovirt-users] Re: Upgraded host, engine now won't boot

[Darrell Budic]

Send it a ctl-alt-delete and see what happens. Possibly try an older kernel at 
the grub boot menu. Could also try stopping it with hosted-engine —vm-stop and 
let HA reboot it, see if it boots or get onto the console quickly and try and 
watch more of the boot.

Ssh and yum upgrade is fine for the OS, although it’s a good idea to enable 
Global HA Maintenance first so the HA watchdogs don’t reboot it in the middle 
of that. After that, run “engine-setup” again, at least if there are new ovirt 
engine updates to be done. Then disable Global HA Maintenance, and run 
"shutdown -h now” to stop the Engine VM (rebooting seems to cause it to exit 
anyway, HA seems to run it as a single execution VM. Or at least in the past, 
it seems to quit anyway on me and shutdown triggered HA faster). Wait a few 
minutes, and HA will respawn it on a new instance and you can log into your 
engine again.
> From: Jim Kusznir 
> Subject: Re: [ovirt-users] Upgraded host, engine now won't boot
> Date: September 3, 2018 at 12:45:22 PM CDT
> To: Darrell Budic
> Cc: users
> 
> Thanks to Jayme who pointed me to the --add-console-password hosted-engine 
> command to set a password for vnc.  Using that, I see only the single line:
> 
> Probing EDD (edd=off to disable)... ok
> 
> --Jim
> 
> On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir  > wrote:
> Is there a way to get a graphical console on boot of the engine vm so I can 
> see what's causing the failure to boot?
> 
> On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir  > wrote:
> Thanks; I guess I didn't mention that I started there.
> 
> The virsh list shows it in state running, and gluster is showing fully online 
> and healed.  However, I cannot bring up a console of the engine VM to see why 
> its not booting, even though it shows in running state.
> 
> In any case, the hosts and engine were running happily.  I applied the latest 
> updates on the host, and the engine went unstable.  I thought, Ok, maybe 
> there's an update to ovirt that also needs to be applied to the engine, so I 
> ssh'ed in and ran yum update (never did find clear instructions on how one is 
> supposed to maintain the engine, but I did see that listed online).  A while 
> later, it reset and never booted again.
> 
> -JIm
> 
> On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic  > wrote:
> It’s definitely not starting, you’ll have to see if you can figure out why. A 
> couple things to try:
> 
> - Check "virsh list" and see if it’s running, or paused for storage. (google 
> "virsh saslpasswd2 
> ”
>  if you need to add a user to do this with, it’s per host)
> -  It’s hyper converged, so check your gluster volume for healing and/or 
> split brains and wait/resolve those.
> - check “gluster peer status” and on each host and make sure your gluster 
> hosts are all talking. I’ve seen an upgrade screwup the firewall, easy fix is 
> to add a rule to allow the hosts to talk to each other on your gluster 
> network, no questions asked (-j ACCEPT, no port, etc).
> 
> Good luck!
> 
>> From: Jim Kusznir mailto:j...@palousetech.com>>
>> Subject: [ovirt-users] Upgraded host, engine now won't boot
>> Date: September 1, 2018 at 8:38:12 PM CDT
>> To: users
>> 
>> Hello:
>> 
>> I saw that there were updates to my ovirt-4.2 3 node hyperconverged system, 
>> so I proceeded to apply them the usual way through the UI.
>> 
>> At one point, the hosted engine was migrated to one of the upgraded hosts, 
>> and then went "unstable" on me.  Now, the hosted engine appears to be 
>> crashed:  It gets powered up, but it never boots up to the point where it 
>> responds to pings or allows logins.  After a while, the hosted engine shows 
>> status (via console "hosted-engine --vm-status" command) "Powering Down".  
>> It stays there for a long time.
>> 
>> I tried forcing a poweroff then powering it on, but again, it never gets up 
>> to where it will respond to pings.  --vm-status shows bad health, but up.
>> 
>> I tried running the hosted-engine --console command, but got:
>> 
>> [root@ovirt1 ~]# hosted-engine --console
>> The engine VM is running on this host
>> Connected to domain HostedEngine
>> Escape character is ^]
>> error: internal error: cannot find character device 
>> 
>> [root@ovirt1 ~]# 
>> 
>> 
>> I tried to run the hosted-engine --upgrade-appliance command, but it hangs 
>> at obtaining certificate (understandably, as the hosted-engine is not up).
>> 
>> How do i recover from this?  And what caused this?
>> 
>> --Jim
>> ___
>> Users mailing list -- users@ovirt.org 
>> To unsubscribe send an email to users-le...@ovirt.org 
>> 
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ 
>> 
>> oVirt Code of Conduct: 
>> 

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

Thanks to Jayme who pointed me to the --add-console-password hosted-engine
command to set a password for vnc.  Using that, I see only the single line:

Probing EDD (edd=off to disable)... ok

--Jim

On Mon, Sep 3, 2018 at 10:26 AM, Jim Kusznir  wrote:

> Is there a way to get a graphical console on boot of the engine vm so I
> can see what's causing the failure to boot?
>
> On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir  wrote:
>
>> Thanks; I guess I didn't mention that I started there.
>>
>> The virsh list shows it in state running, and gluster is showing fully
>> online and healed.  However, I cannot bring up a console of the engine VM
>> to see why its not booting, even though it shows in running state.
>>
>> In any case, the hosts and engine were running happily.  I applied the
>> latest updates on the host, and the engine went unstable.  I thought, Ok,
>> maybe there's an update to ovirt that also needs to be applied to the
>> engine, so I ssh'ed in and ran yum update (never did find clear
>> instructions on how one is supposed to maintain the engine, but I did see
>> that listed online).  A while later, it reset and never booted again.
>>
>> -JIm
>>
>> On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic 
>> wrote:
>>
>>> It’s definitely not starting, you’ll have to see if you can figure out
>>> why. A couple things to try:
>>>
>>> - Check "virsh list" and see if it’s running, or paused for storage.
>>> (google "virsh saslpasswd2
>>> ”
>>> if you need to add a user to do this with, it’s per host)
>>> -  It’s hyper converged, so check your gluster volume for healing and/or
>>> split brains and wait/resolve those.
>>> - check “gluster peer status” and on each host and make sure your
>>> gluster hosts are all talking. I’ve seen an upgrade screwup the firewall,
>>> easy fix is to add a rule to allow the hosts to talk to each other on your
>>> gluster network, no questions asked (-j ACCEPT, no port, etc).
>>>
>>> Good luck!
>>>
>>> --
>>> *From:* Jim Kusznir 
>>> *Subject:* [ovirt-users] Upgraded host, engine now won't boot
>>> *Date:* September 1, 2018 at 8:38:12 PM CDT
>>> *To:* users
>>>
>>> Hello:
>>>
>>> I saw that there were updates to my ovirt-4.2 3 node hyperconverged
>>> system, so I proceeded to apply them the usual way through the UI.
>>>
>>> At one point, the hosted engine was migrated to one of the upgraded
>>> hosts, and then went "unstable" on me.  Now, the hosted engine appears to
>>> be crashed:  It gets powered up, but it never boots up to the point where
>>> it responds to pings or allows logins.  After a while, the hosted engine
>>> shows status (via console "hosted-engine --vm-status" command) "Powering
>>> Down".  It stays there for a long time.
>>>
>>> I tried forcing a poweroff then powering it on, but again, it never gets
>>> up to where it will respond to pings.  --vm-status shows bad health, but up.
>>>
>>> I tried running the hosted-engine --console command, but got:
>>>
>>> [root@ovirt1 ~]# hosted-engine --console
>>> The engine VM is running on this host
>>> Connected to domain HostedEngine
>>> Escape character is ^]
>>> error: internal error: cannot find character device 
>>>
>>> [root@ovirt1 ~]#
>>>
>>>
>>> I tried to run the hosted-engine --upgrade-appliance command, but it
>>> hangs at obtaining certificate (understandably, as the hosted-engine is not
>>> up).
>>>
>>> How do i recover from this?  And what caused this?
>>>
>>> --Jim
>>> ___
>>> Users mailing list -- users@ovirt.org
>>> To unsubscribe send an email to users-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct: https://www.ovirt.org/communit
>>> y/about/community-guidelines/
>>> List Archives: https://lists.ovirt.org/archiv
>>> es/list/users@ovirt.org/message/XBNOOF4OA5C5AFGCT3KGUPUTRSOLIPXX/
>>>
>>>
>>>
>>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C62LXGEOGRDWCEZ6XWN3YUSGS32IPROS/

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

Is there a way to get a graphical console on boot of the engine vm so I can
see what's causing the failure to boot?

On Mon, Sep 3, 2018 at 10:23 AM, Jim Kusznir  wrote:

> Thanks; I guess I didn't mention that I started there.
>
> The virsh list shows it in state running, and gluster is showing fully
> online and healed.  However, I cannot bring up a console of the engine VM
> to see why its not booting, even though it shows in running state.
>
> In any case, the hosts and engine were running happily.  I applied the
> latest updates on the host, and the engine went unstable.  I thought, Ok,
> maybe there's an update to ovirt that also needs to be applied to the
> engine, so I ssh'ed in and ran yum update (never did find clear
> instructions on how one is supposed to maintain the engine, but I did see
> that listed online).  A while later, it reset and never booted again.
>
> -JIm
>
> On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic 
> wrote:
>
>> It’s definitely not starting, you’ll have to see if you can figure out
>> why. A couple things to try:
>>
>> - Check "virsh list" and see if it’s running, or paused for storage.
>> (google "virsh saslpasswd2
>> ”
>> if you need to add a user to do this with, it’s per host)
>> -  It’s hyper converged, so check your gluster volume for healing and/or
>> split brains and wait/resolve those.
>> - check “gluster peer status” and on each host and make sure your gluster
>> hosts are all talking. I’ve seen an upgrade screwup the firewall, easy fix
>> is to add a rule to allow the hosts to talk to each other on your gluster
>> network, no questions asked (-j ACCEPT, no port, etc).
>>
>> Good luck!
>>
>> --
>> *From:* Jim Kusznir 
>> *Subject:* [ovirt-users] Upgraded host, engine now won't boot
>> *Date:* September 1, 2018 at 8:38:12 PM CDT
>> *To:* users
>>
>> Hello:
>>
>> I saw that there were updates to my ovirt-4.2 3 node hyperconverged
>> system, so I proceeded to apply them the usual way through the UI.
>>
>> At one point, the hosted engine was migrated to one of the upgraded
>> hosts, and then went "unstable" on me.  Now, the hosted engine appears to
>> be crashed:  It gets powered up, but it never boots up to the point where
>> it responds to pings or allows logins.  After a while, the hosted engine
>> shows status (via console "hosted-engine --vm-status" command) "Powering
>> Down".  It stays there for a long time.
>>
>> I tried forcing a poweroff then powering it on, but again, it never gets
>> up to where it will respond to pings.  --vm-status shows bad health, but up.
>>
>> I tried running the hosted-engine --console command, but got:
>>
>> [root@ovirt1 ~]# hosted-engine --console
>> The engine VM is running on this host
>> Connected to domain HostedEngine
>> Escape character is ^]
>> error: internal error: cannot find character device 
>>
>> [root@ovirt1 ~]#
>>
>>
>> I tried to run the hosted-engine --upgrade-appliance command, but it
>> hangs at obtaining certificate (understandably, as the hosted-engine is not
>> up).
>>
>> How do i recover from this?  And what caused this?
>>
>> --Jim
>> ___
>> Users mailing list -- users@ovirt.org
>> To unsubscribe send an email to users-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct: https://www.ovirt.org/communit
>> y/about/community-guidelines/
>> List Archives: https://lists.ovirt.org/archiv
>> es/list/users@ovirt.org/message/XBNOOF4OA5C5AFGCT3KGUPUTRSOLIPXX/
>>
>>
>>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WKPMXVUYM5AAD7KYAYLB4DJ4NYGKXZFE/

[ovirt-users] Re: Upgraded host, engine now won't boot

[Jim Kusznir]

Thanks; I guess I didn't mention that I started there.

The virsh list shows it in state running, and gluster is showing fully
online and healed.  However, I cannot bring up a console of the engine VM
to see why its not booting, even though it shows in running state.

In any case, the hosts and engine were running happily.  I applied the
latest updates on the host, and the engine went unstable.  I thought, Ok,
maybe there's an update to ovirt that also needs to be applied to the
engine, so I ssh'ed in and ran yum update (never did find clear
instructions on how one is supposed to maintain the engine, but I did see
that listed online).  A while later, it reset and never booted again.

-JIm

On Sun, Sep 2, 2018 at 4:28 PM, Darrell Budic 
wrote:

> It’s definitely not starting, you’ll have to see if you can figure out
> why. A couple things to try:
>
> - Check "virsh list" and see if it’s running, or paused for storage.
> (google "virsh saslpasswd2
> ”
> if you need to add a user to do this with, it’s per host)
> -  It’s hyper converged, so check your gluster volume for healing and/or
> split brains and wait/resolve those.
> - check “gluster peer status” and on each host and make sure your gluster
> hosts are all talking. I’ve seen an upgrade screwup the firewall, easy fix
> is to add a rule to allow the hosts to talk to each other on your gluster
> network, no questions asked (-j ACCEPT, no port, etc).
>
> Good luck!
>
> --
> *From:* Jim Kusznir 
> *Subject:* [ovirt-users] Upgraded host, engine now won't boot
> *Date:* September 1, 2018 at 8:38:12 PM CDT
> *To:* users
>
> Hello:
>
> I saw that there were updates to my ovirt-4.2 3 node hyperconverged
> system, so I proceeded to apply them the usual way through the UI.
>
> At one point, the hosted engine was migrated to one of the upgraded hosts,
> and then went "unstable" on me.  Now, the hosted engine appears to be
> crashed:  It gets powered up, but it never boots up to the point where it
> responds to pings or allows logins.  After a while, the hosted engine shows
> status (via console "hosted-engine --vm-status" command) "Powering Down".
> It stays there for a long time.
>
> I tried forcing a poweroff then powering it on, but again, it never gets
> up to where it will respond to pings.  --vm-status shows bad health, but up.
>
> I tried running the hosted-engine --console command, but got:
>
> [root@ovirt1 ~]# hosted-engine --console
> The engine VM is running on this host
> Connected to domain HostedEngine
> Escape character is ^]
> error: internal error: cannot find character device 
>
> [root@ovirt1 ~]#
>
>
> I tried to run the hosted-engine --upgrade-appliance command, but it hangs
> at obtaining certificate (understandably, as the hosted-engine is not up).
>
> How do i recover from this?  And what caused this?
>
> --Jim
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
> guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
> message/XBNOOF4OA5C5AFGCT3KGUPUTRSOLIPXX/
>
>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/25AE7XZBRFCWG3HKZBGAC2KBXDZLBOC2/

[ovirt-users] Re: Upgraded host, engine now won't boot

[Darrell Budic]

It’s definitely not starting, you’ll have to see if you can figure out why. A 
couple things to try:

- Check "virsh list" and see if it’s running, or paused for storage. (google 
"virsh saslpasswd2 
”
 if you need to add a user to do this with, it’s per host)
-  It’s hyper converged, so check your gluster volume for healing and/or split 
brains and wait/resolve those.
- check “gluster peer status” and on each host and make sure your gluster hosts 
are all talking. I’ve seen an upgrade screwup the firewall, easy fix is to add 
a rule to allow the hosts to talk to each other on your gluster network, no 
questions asked (-j ACCEPT, no port, etc).

Good luck!

> From: Jim Kusznir 
> Subject: [ovirt-users] Upgraded host, engine now won't boot
> Date: September 1, 2018 at 8:38:12 PM CDT
> To: users
> 
> Hello:
> 
> I saw that there were updates to my ovirt-4.2 3 node hyperconverged system, 
> so I proceeded to apply them the usual way through the UI.
> 
> At one point, the hosted engine was migrated to one of the upgraded hosts, 
> and then went "unstable" on me.  Now, the hosted engine appears to be 
> crashed:  It gets powered up, but it never boots up to the point where it 
> responds to pings or allows logins.  After a while, the hosted engine shows 
> status (via console "hosted-engine --vm-status" command) "Powering Down".  It 
> stays there for a long time.
> 
> I tried forcing a poweroff then powering it on, but again, it never gets up 
> to where it will respond to pings.  --vm-status shows bad health, but up.
> 
> I tried running the hosted-engine --console command, but got:
> 
> [root@ovirt1 ~]# hosted-engine --console
> The engine VM is running on this host
> Connected to domain HostedEngine
> Escape character is ^]
> error: internal error: cannot find character device 
> 
> [root@ovirt1 ~]# 
> 
> 
> I tried to run the hosted-engine --upgrade-appliance command, but it hangs at 
> obtaining certificate (understandably, as the hosted-engine is not up).
> 
> How do i recover from this?  And what caused this?
> 
> --Jim
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/XBNOOF4OA5C5AFGCT3KGUPUTRSOLIPXX/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FRGKRZF4G4S2HNN5TNN7DOPEODJBAQSD/