On Thu, 21 Jul 2016 14:43:50 -0400 Robert wrote: RS> So after some debugging with Simone on irc, we've determined that the issue RS> is the agent timing out trying to communicate with the broker. The problem RS> is that we have no idea why.
So more detail attached. The agent is sending:
MainThread::hosted_engine::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine
::(start_monitoring) Processing engine state
<ovirt_hosted_engine_ha.agent.states.ReinitializeFSM object at 0x15d8c30>
MainThread::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink
::(notify) Trying: notify time=1469129518.85 type=state_transition
detail=StartState-ReinitializeFSM hostname='poseidon.netsec'
MainThread::brokerlink::273::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink
::(_communicate) Sending request: notify time=1469129518.85
type=state_transition detail=StartState-ReinitializeFSM
hostname='poseidon.netsec'
Which the broker sees:
Thread-1::util::69::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
::(socket_readline) socket_readline in blocking mode
Thread-1::listener::163::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
::(handle) Input: notify time=1469129518.85 type=state_transition
detail=StartState-ReinitializeFSM hostname='poseidon.netsec'
It then refreshes the local config file:
Thread-1::config::251::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) Reading 'broker.conf' from
'/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7
Which succeeds:
Thread-1::config::271::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) Writing to
'/var/lib/ovirt-hosted-engine-ha/broker.conf'
Thread-1::config::278::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) local conf file was correctly written
And then .... nothing. It just hangs. Nothing more is logged Thread-1.
Robert
--
Senior Software Engineer @ Parsons
Robert
--
Senior Software Engineer @ Parsons
****************************************************************************************************************************************
MainThread::DEBUG::2016-07-21 15:31:58,847::hosted_engine::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Processing engine state <ovirt_hosted_engine_ha.agent.states.ReinitializeFSM object at 0x15d8c30>
MainThread::INFO::2016-07-21 15:31:58,847::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec'
MainThread::DEBUG::2016-07-21 15:31:58,847::brokerlink::273::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Sending request: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec'
MainThread::DEBUG::2016-07-21 15:31:58,848::util::77::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(socket_readline) socket_readline with 30.0 seconds timeout
MainThread::DEBUG::2016-07-21 15:32:28,866::util::88::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(socket_readline) Connection timeout while reading from socket
MainThread::ERROR::2016-07-21 15:32:28,867::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection timed out
MainThread::DEBUG::2016-07-21 15:32:28,867::brokerlink::86::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(disconnect) Closing connection to ha-broker
MainThread::ERROR::2016-07-21 15:32:28,867::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to start monitor state_transition, options {'hostname': 'poseidon.netsec'}: Connection timed out' - trying to restart agent
****************************************************************************************************************************************
****************************************************************************************************************************************************************
Thread-1::util::69::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
::(socket_readline) socket_readline in blocking mode
Thread-1::listener::163::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
::(handle) Input: notify time=1469129518.85 type=state_transition detail=StartState-ReinitializeFSM hostname='poseidon.netsec'
Thread-1::listener::238::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler
::(_dispatch) Request type notify from 139793244509952
Thread-1::notifications::46::ovirt_hosted_engine_ha.broker.notifications.Notifications
::(notify) nofity: {'hostname': 'poseidon.netsec', 'type': 'state_transition', 'detail': 'StartState-ReinitializeFSM', 'time': '1469129518.85'}
Thread-1::config::251::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) Reading 'broker.conf' from '/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7c70e-d6ca-482f-b414-d458f7f5f9de'
Thread-1::heconflib::69::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) executing: 'sudo -u vdsm dd if=/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7c70e-d6ca-482f-b414-d458f7f5f9de bs=4k'
Thread-1::heconflib::70::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) executing: 'tar -tvf -'
Thread-1::heconflib::88::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) stdout: -rw-r--r-- 0/0 7 1969-12-31 19:00 version
-rw-r--r-- 0/0 2572 1969-12-31 19:00 fhanswers.conf
-rw-r--r-- 0/0 861 1969-12-31 19:00 hosted-engine.conf
-rw-r--r-- 0/0 182 1969-12-31 19:00 broker.conf
-rw-r--r-- 0/0 1315 1969-12-31 19:00 vm.conf
Thread-1::heconflib::89::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) stderr:
Thread-1::heconflib::138::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(extractConfFile) extracting 'broker.conf' from '/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7c70e-d6ca-482f-b414-d458f7f5f9de'
Thread-1::heconflib::69::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) executing: 'sudo -u vdsm dd if=/rhev/data-center/mnt/ovirt-nfs.netsec:_ovirt_hosted-engine/2daba0ab-2b3d-4026-bcfc-1cd071c30038/images/a04a45b9-e780-4104-ad4b-d5901a5490c4/34a7c70e-d6ca-482f-b414-d458f7f5f9de bs=4k'
Thread-1::heconflib::70::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) executing: 'tar -xOf - broker.conf'
Thread-1::heconflib::88::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) stdout: [email]
smtp-server = localhost
smtp-port = 25
destination-emails = root@localhost
source-email = root@localhost
[notify]
state_transition = maintenance|start|stop|migrate|up|down
Thread-1::heconflib::89::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(_dd_pipe_tar) stderr:
Thread-1::config::271::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) Writing to '/var/lib/ovirt-hosted-engine-ha/broker.conf'
Thread-1::config::278::ovirt_hosted_engine_ha.broker.notifications.Notifications.config
::(refresh_local_conf_file) local conf file was correctly written
****************************************************************************************************************************************************************
pgpNvG3wYMFqR.pgp
Description: OpenPGP digital signature
_______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
