[ovirt-users] Re: Hosted engine not starting after 4.3 Upgrade - cannot find OVF_STORE
Thank you!
The ownership of the volume file had changed to root:root. I changed it back
to vdsm:kvm and the hosted engine started.
For anyone else who runs in to this, the file was in:
/rhev/data-center/mnt/glusterSD/ovirtnode-02:_vmstore/79376c46-b80c-4c44-bbb1-80c0714a4b52/images/48ee766b-185d-4928-a046-b048d65af2a6
The errors in vdsm.log that pointed to this:
2019-03-06 08:16:24,470-0500 INFO (jsonrpc/4) [vdsm.api] START
getVolumeInfo(sdUUID=u'79376c46-b80c-4c44-bbb1-80c0714a4b52',
spUUID=u'----',
imgUUID=u'48ee766b-185d-4928-a046-b048d65af2a6',
volUUID=u'687e9c0d-e988-4f76-89ff-931685acdf76', options=None) from=::1,37228,
task_id=8170eec2-b3f7-488c-adda-3f1d9b1d0c57 (api:48)
2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [vdsm.api] FINISH getVolumeInfo
error=Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)
from=::1,37228, task_id=8170eec2-b3f7-488c-adda-3f1d9b1d0c57 (api:52)
2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [storage.TaskManager.Task]
(Task='8170eec2-b3f7-488c-adda-3f1d9b1d0c57') aborting: Task is aborted:
"Volume does not exist: (u'687e9c0d-e988-4f76-89ff-931685acdf76',)" - code 201
(task:1181)
2019-03-06 08:16:24,472-0500 INFO (jsonrpc/4) [storage.Dispatcher] FINISH
getVolumeInfo error=Volume does not exist:
(u'687e9c0d-e988-4f76-89ff-931685acdf76',) (dispatcher:81)
From: Jayme
Sent: March 6, 2019 6:51 AM
To: Shawn Southern
Cc: users@ovirt.org
Subject: Re: [ovirt-users] Hosted engine not starting after 4.3 Upgrade -
cannot find OVF_STORE
It sure if this is the same bug I hit but check ownership of the cam images.
There’s a bug in 4.3 upgrade that changes ownership to root and causes vms to
not start until you change back to vdsm
On Wed, Mar 6, 2019 at 4:57 AM Shawn Southern
mailto:shawn.south...@entegrus.com>> wrote:
After running 'hosted-engine --vm-start', the status of the hosted engine VM is:
conf_on_shared_storage : True
Status up-to-date : True
Hostname : ovirtnode-01
Host ID: 3
Engine status : {"reason": "bad vm status", "health":
"bad", "vm": "down_unexpected", "detail": "Down"}
Score : 0
stopped: False
Local maintenance : False
crc32 : 7e3db850
local_conf_timestamp : 3509
Host timestamp : 3508
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=3508 (Tue Mar 5 16:03:30 2019)
host-id=3
score=0
vm_conf_refresh_time=3509 (Tue Mar 5 16:03:31 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Wed Dec 31 20:05:37 1969
The /var/log/libvirt/qemu/HostedEngine.log has no entries since the hosted
engine VM was rebooted.
/var/log/ovirt-hosted-engine-ha/agent.log:
MainThread::ERROR::2019-03-05
16:07:31,916::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
Failed scanning for OVF_STORE due to Command Volume.getInfo with args
{'storagepoolID': '----', 'storagedomainID':
'79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID':
u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID':
u'48ee766b-185d-4928-a046-b048d65af2a6'} failed:
(code=201, message=Volume does not exist:
(u'687e9c0d-e988-4f76-89ff-931685acdf76',))
MainThread::ERROR::2019-03-05
16:07:31,916::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
Please ensure you already added your first data domain for regular VMs
MainThread::INFO::2019-03-05
16:07:31,971::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
Current state EngineUnexpectedlyDown (score: 0)
MainThread::ERROR::2019-03-05
16:07:42,304::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
Failed scanning for OVF_STORE due to Command Volume.getInfo with args
{'storagepoolID': '----', 'storagedomainID':
'79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID':
u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID':
u'48ee766b-185d-4928-a046-b048d65af2a6'} failed:
(code=201, message=Volume does not exist:
(u'687e9c0d-e988-4f76-89ff-931685acdf76',))
MainThread::ERROR::2019-03-05
16:07:42,305::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
Ple
[ovirt-users] Re: Hosted engine not starting after 4.3 Upgrade - cannot find OVF_STORE
It sure if this is the same bug I hit but check ownership of the cam
images. There’s a bug in 4.3 upgrade that changes ownership to root and
causes vms to not start until you change back to vdsm
On Wed, Mar 6, 2019 at 4:57 AM Shawn Southern
wrote:
> After running 'hosted-engine --vm-start', the status of the hosted engine
> VM is:
>
> conf_on_shared_storage : True
> Status up-to-date : True
> Hostname : ovirtnode-01
> Host ID: 3
> Engine status : {"reason": "bad vm status", "health":
> "bad", "vm": "down_unexpected", "detail": "Down"}
> Score : 0
> stopped: False
> Local maintenance : False
> crc32 : 7e3db850
> local_conf_timestamp : 3509
> Host timestamp : 3508
> Extra metadata (valid at timestamp):
> metadata_parse_version=1
> metadata_feature_version=1
> timestamp=3508 (Tue Mar 5 16:03:30 2019)
> host-id=3
> score=0
> vm_conf_refresh_time=3509 (Tue Mar 5 16:03:31 2019)
> conf_on_shared_storage=True
> maintenance=False
> state=EngineUnexpectedlyDown
> stopped=False
> timeout=Wed Dec 31 20:05:37 1969
>
>
> The /var/log/libvirt/qemu/HostedEngine.log has no entries since the hosted
> engine VM was rebooted.
>
> /var/log/ovirt-hosted-engine-ha/agent.log:
> MainThread::ERROR::2019-03-05
> 16:07:31,916::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Failed scanning for OVF_STORE due to Command Volume.getInfo with args
> {'storagepoolID': '----',
> 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID':
> u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID':
> u'48ee766b-185d-4928-a046-b048d65af2a6'} failed:
> (code=201, message=Volume does not exist:
> (u'687e9c0d-e988-4f76-89ff-931685acdf76',))
> MainThread::ERROR::2019-03-05
> 16:07:31,916::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
> Please ensure you already added your first data domain for regular VMs
> MainThread::INFO::2019-03-05
> 16:07:31,971::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
> Current state EngineUnexpectedlyDown (score: 0)
> MainThread::ERROR::2019-03-05
> 16:07:42,304::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Failed scanning for OVF_STORE due to Command Volume.getInfo with args
> {'storagepoolID': '----',
> 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID':
> u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID':
> u'48ee766b-185d-4928-a046-b048d65af2a6'} failed:
> (code=201, message=Volume does not exist:
> (u'687e9c0d-e988-4f76-89ff-931685acdf76',))
> MainThread::ERROR::2019-03-05
> 16:07:42,305::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
> Please ensure you already added your first data domain for regular VMs
> MainThread::INFO::2019-03-05
> 16:07:42,365::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
> Current state EngineUnexpectedlyDown (score: 0)
> MainThread::ERROR::2019-03-05
> 16:07:51,791::config_ovf::42::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Failed scanning for OVF_STORE due to Command Volume.getInfo with args
> {'storagepoolID': '----',
> 'storagedomainID': '79376c46-b80c-4c44-bbb1-80c0714a4b52', 'volumeID':
> u'687e9c0d-e988-4f76-89ff-931685acdf76', 'imageID':
> u'48ee766b-185d-4928-a046-b048d65af2a6'} failed:
> (code=201, message=Volume does not exist:
> (u'687e9c0d-e988-4f76-89ff-931685acdf76',))
> MainThread::ERROR::2019-03-05
> 16:07:51,792::config_ovf::84::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store)
> Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
> Please ensure you already added your first data domain for regular VMs
> MainThread::INFO::2019-03-05
> 16:07:51,850::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
> Current state EngineUnexpectedlyDown (score: 0)
> MainThread::INFO::2019-03-05
> 16:08:01,868::states::684::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine down, local host does not have best score
> MainThread::ERROR::2019-03-05
>
