This happened to me again, started last night so it was almost a week from the last restart. System was not out of memory, a bit low, and it may have been churning buffers or java GC, I’m on vacation and didn’t dig into it very far. Restarted the engine and it’s happy. DWH was still working, but web interface was a bit slow before the restart. This was 4.1.3 now. Added some ram to the Hosted Engine, but looks like I need to restart it and will probably wait until I’m back for that.
> On Jul 18, 2017, at 9:22 AM, Darrell Budic <bu...@onholyground.com> wrote: > > I had some of this going on recently under 4.1.2, started with one or two > warning messages, then a flood of them. Did the upgrade to 4.1.3 and haven’t > seen it yet, but it’s only been a few days so far. A java process was > consuming much CPU, and the DataWarehouse appears to not be collecting data > (evidenced by a blank dashboard). My DWH has since recovered as well. > > I forgot to check, but suspect I was low/out of memory on my engine VM, it’s > an old one with only 6G allocated currently. Watching for this to happen > again, and will confirm RAM utilization and bump up appropriately if it looks > like it’s starved for RAM. > > >> On Jul 18, 2017, at 5:45 AM, Christophe TREFOIS <christophe.tref...@uni.lu >> <mailto:christophe.tref...@uni.lu>> wrote: >> >> I have the same as you on 4.1.0 >> >> EngineBadHealth-EngineUp 1 minute later. Sometimes 20 times per day, mostly >> on weekends. >> >> Cheers, >> -- >> >> Dr Christophe Trefois, Dipl.-Ing. >> Technical Specialist / Post-Doc >> >> UNIVERSITÉ DU LUXEMBOURG >> >> LUXEMBOURG CENTRE FOR SYSTEMS BIOMEDICINE >> Campus Belval | House of Biomedicine >> 6, avenue du Swing >> L-4367 Belvaux >> T: +352 46 66 44 6124 >> F: +352 46 66 44 6949 >> http://www.uni.lu/lcsb <http://www.uni.lu/lcsb> >> <https://www.facebook.com/trefex> <https://twitter.com/Trefex> >> <https://plus.google.com/+ChristopheTrefois/> >> <https://www.linkedin.com/in/trefoischristophe> <http://skype:Trefex?call> >> >> ---- >> This message is confidential and may contain privileged information. >> It is intended for the named recipient only. >> If you receive it in error please notify me and permanently delete the >> original message and any copies. >> ---- >> >> >>> On 17 Jul 2017, at 17:35, Jim Kusznir <j...@palousetech.com >>> <mailto:j...@palousetech.com>> wrote: >>> >>> Ok, I've been ignoring this for a long time as the logs were so verbose and >>> didn't show anything I could identify as usable debug info. Recently one >>> of my ovirt hosts (currently NOT running the main engine, but a candidate) >>> was cycling as much as 40 times a day between "EngineUpBadHealth and >>> EngineUp". Here's the log snippit. I included some time before and after >>> if that's helpful. In this case, I got an email about bad health at 8:15 >>> and a restore (engine up) at 8:16. I see where the messages are sent, but >>> I don't see any explanation as to why / what the problem is. >>> >>> BTW: 192.168.8.11 is this computer's physical IP; 192.168.8.12 is the >>> computer currently running the engine. Both are also hosting the gluster >>> store (eg, I have 3 hosts, all are participating in the gluster replica >>> 2+arbitrator). >>> >>> I'd appreciate it if someone could shed some light on why this keeps >>> happening! >>> >>> --Jim >>> ---------------- >>> >>> MainThread::INFO::2017-07-17 >>> 08:12:06,230::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:12:06,230::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:08,877::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:12:09,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:12:09,925::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:10,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:12:10,696::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:12:10,704::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:10,705::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >>> Engine vm running on localhost >>> MainThread::INFO::2017-07-17 >>> 08:12:10,714::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:12:14,426::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:12:14,470::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:12:19,648::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:12:19,900::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:12:20,298::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:12:20,298::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:12:24,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:12:24,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:12:24,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:26,895::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:12:27,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:12:27,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:28,240::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:12:28,564::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:12:28,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:34,574::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUp (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:12:34,575::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:12:44,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:12:44,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:47,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:12:48,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:12:48,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:49,128::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:12:49,552::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:12:49,559::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:12:49,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:12:52,152::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:12:52,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:12:58,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:12:58,406::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:12:58,805::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:12:58,806::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:13:02,662::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:13:02,662::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:13:02,663::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:13:06,057::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:13:06,646::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:07,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:13:07,595::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:13:07,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:13,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUp (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:13:13,644::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:13:23,760::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:13:23,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:26,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:13:27,129::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:13:27,770::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:28,154::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:13:28,603::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:13:28,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:28,613::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >>> Engine vm running on localhost >>> MainThread::INFO::2017-07-17 >>> 08:13:28,622::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:13:31,357::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:13:31,420::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:13:37,594::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:13:37,848::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:13:38,300::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:13:38,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:13:41,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:13:41,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:13:41,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:44,478::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:13:44,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:13:45,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:45,703::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:13:46,010::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:13:46,014::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:13:51,170::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUp (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:13:51,170::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:14:01,282::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:01,282::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:04,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:14:05,394::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:14:05,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:06,277::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:14:06,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:14:06,632::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:06,637::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:14:09,448::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:14:09,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:14:14,997::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:14:15,267::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:15,670::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:14:15,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:14:20,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:14:20,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:20,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:23,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:14:24,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:14:24,604::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:24,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:14:25,391::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:14:25,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:30,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUp (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:14:30,597::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:14:40,749::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:40,749::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:43,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:14:44,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:14:44,868::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:45,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:14:45,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:14:45,676::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:14:45,677::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >>> Engine vm running on localhost >>> MainThread::INFO::2017-07-17 >>> 08:14:45,684::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:14:49,208::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:14:49,246::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:14:54,226::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:14:54,453::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:54,835::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:14:54,835::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:14:58,377::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:14:58,377::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:14:58,377::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:01,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:15:01,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:15:02,386::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:02,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:15:03,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:15:03,299::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:09,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUp (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:15:09,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:15:19,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:15:19,413::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:22,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:15:22,913::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:15:23,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:23,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:15:24,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:15:24,309::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:24,318::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) >>> Trying: notify time=1500304524.32 type=state_transition >>> detail=EngineUp-EngineUpBadHealth hostname='ovirt1.nwfiber.com >>> <http://ovirt1.nwfiber.com/>' >>> MainThread::INFO::2017-07-17 >>> 08:15:24,450::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) >>> Success, was notification of state_transition (EngineUp-EngineUpBadHealth) >>> sent? sent >>> MainThread::INFO::2017-07-17 >>> 08:15:24,451::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:15:27,085::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:15:27,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:15:33,220::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:15:33,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:15:34,134::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:15:34,134::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:15:37,991::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Refreshing vm.conf >>> MainThread::INFO::2017-07-17 >>> 08:15:37,991::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:15:37,991::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:40,745::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:15:41,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:15:42,054::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:42,563::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:15:43,062::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:15:43,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:15:49,373::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Current state EngineUpBadHealth (score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:15:49,374::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) >>> Best remote host ovirt2.nwfiber.com <http://ovirt2.nwfiber.com/> (id: 2, >>> score: 3400) >>> MainThread::INFO::2017-07-17 >>> 08:15:59,492::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) >>> Reloading vm.conf from the shared storage domain >>> MainThread::INFO::2017-07-17 >>> 08:15:59,492::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Trying to get a fresher copy of vm configuration from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:16:02,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:e10c90a5-4d9c-4e18-b6f7-ae8f0cdf4f57, >>> volUUID:a9754d40-eda1-44d7-ac92-76a228f9f1ac >>> MainThread::INFO::2017-07-17 >>> 08:16:03,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) >>> Found OVF_STORE: imgUUID:f22829ab-9fd5-415a-9a8f-809d3f7887d4, >>> volUUID:9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> MainThread::INFO::2017-07-17 >>> 08:16:03,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> Extracting Engine VM OVF from the OVF_STORE >>> MainThread::INFO::2017-07-17 >>> 08:16:04,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) >>> OVF_STORE volume path: >>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine/c0acdefb-7d16-48ec-9d76-659b8fe33e2a/images/f22829ab-9fd5-415a-9a8f-809d3f7887d4/9f4760ee-119c-412a-a1e8-49e73e6ba929 >>> >>> MainThread::INFO::2017-07-17 >>> 08:16:04,680::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Found an OVF for HE VM, trying to convert >>> MainThread::INFO::2017-07-17 >>> 08:16:04,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) >>> Got vm.conf from OVF_STORE >>> MainThread::ERROR::2017-07-17 >>> 08:16:04,690::states::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >>> Engine VM has bad health status, timeout in 300 seconds >>> MainThread::INFO::2017-07-17 >>> 08:16:04,691::states::426::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >>> Engine vm running on localhost >>> MainThread::INFO::2017-07-17 >>> 08:16:04,700::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) >>> Timeout cleared while transitioning <class >>> 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class >>> 'ovirt_hosted_engine_ha.agent.states.EngineUp'> >>> MainThread::INFO::2017-07-17 >>> 08:16:04,710::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) >>> Trying: notify time=1500304564.71 type=state_transition >>> detail=EngineUpBadHealth-EngineUp hostname='ovirt1.nwfiber.com >>> <http://ovirt1.nwfiber.com/>' >>> MainThread::INFO::2017-07-17 >>> 08:16:04,798::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) >>> Success, was notification of state_transition (EngineUpBadHealth-EngineUp) >>> sent? sent >>> MainThread::INFO::2017-07-17 >>> 08:16:04,799::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) >>> Initializing VDSM >>> MainThread::INFO::2017-07-17 >>> 08:16:07,435::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Connecting the storage >>> MainThread::INFO::2017-07-17 >>> 08:16:07,491::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:16:13,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Connecting storage server >>> MainThread::INFO::2017-07-17 >>> 08:16:14,131::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) >>> Refreshing the storage domain >>> MainThread::INFO::2017-07-17 >>> 08:16:14,437::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) >>> Preparing images >>> MainThread::INFO::2017-07-17 >>> 08:16:14,438::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) >>> Preparing images >>> >>> >>> On Thu, Mar 30, 2017 at 5:58 AM, Simone Tiraboschi <stira...@redhat.com >>> <mailto:stira...@redhat.com>> wrote: >>> Could you please check your /var/log/ovirt-hosted-engine-ha/agent.log ? >>> >>> On Thu, Mar 30, 2017 at 3:10 AM, Jim Kusznir <j...@palousetech.com >>> <mailto:j...@palousetech.com>> wrote: >>> Hello: >>> >>> I find that I often get random-seeming messages. A lot of them mention >>> "ReintializeFSM", but I also get engine down, engine start, etc. messages. >>> All the time, nothing appears to be happening on the cluster, and I rarely >>> can find anything wrong or any trigger/cause. Is this normal? What causes >>> this (beyond obvious hardware issues / hosts rebooting)? Most of the time >>> when I get these, my cluster is going along smoothly, and nothing (not even >>> administrative access) is interrupted. >>> >>> Could ISP issues cause these messages to be generated? >>> >>> Thanks! >>> --Jim >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org <mailto:Users@ovirt.org> >>> http://lists.ovirt.org/mailman/listinfo/users >>> <http://lists.ovirt.org/mailman/listinfo/users> >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org <mailto:Users@ovirt.org> >>> http://lists.ovirt.org/mailman/listinfo/users >>> <http://lists.ovirt.org/mailman/listinfo/users> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org <mailto:Users@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
