Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-29 Thread Dan Kenigsberg 
On Tue, Jun 28, 2016 at 06:06:39PM +0200, Simone Tiraboschi wrote:
> On Tue, Jun 28, 2016 at 5:24 PM, Dan Kenigsberg  wrote:
> > On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote:
> >> On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg  wrote:
> >> > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:
> >> >> Hi!
> >> >
> >> >
> >> > Thanks for the detailed logging!
> >> >
> >> >>
> >> >> The broker error is:
> >> >>
> >> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> >> >> MainThread::INFO::2016-06-27 
> >> >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> >> >> Success, id 140293563619152
> >> >>
> >> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> >> >> Thread-25::ERROR::2016-06-27 
> >> >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> >> >> Error while serving connection
> >> >> Traceback (most recent call last):
> >> >>   File
> >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> >> >> line 166, in handle
> >> >> data)
> >> >>   File
> >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> >> >> line 299, in _dispatch
> >> >> .set_storage_domain(client, sd_type, **options)
> >> >>   File 
> >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> >> >> line 66, in set_storage_domain
> >> >> self._backends[client].connect()
> >> >>   File 
> >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> >> >> line 400, in connect
> >> >> volUUID=volume.volume_uuid
> >> >>   File 
> >> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> >> >> line 245, in _get_volume_path
> >> >> volUUID
> >> >
> >> > We have two issues here. First is that
> >> > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
> >> > is still using vdscli to contact vdsm, instead of the preferred
> >> > jsonrpccli.
> >>
> >> We already have this one:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1101554
> >>
> > Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient
> >
> > It does not seem to be the same thing; here I'm referring to
> > ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one
> > condition it's the soon-to-be-deprecated xmlrpc.
> 
> ovirt-hosted-engine-setup is already using just API via jsonrpc.
> The issue is that ovirt-hosted-engine-ha somewhere uses vdsClient
> which AFAIK internally still uses xmlrpc

It's not a general "somewhere", I gave the specific link ;-)
https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-29 Thread Stefano Danzi 


HI to All!!

Il 28/06/2016 17.24, Stefano Danzi ha scritto:

[CUT]
We have two issues here. First is that
https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271 


is still using vdscli to contact vdsm, instead of the preferred
jsonrpccli.

The second is that vdscli.connect's heuristic ends up reading the local
server address from vdsm config, where it finds the default ipv6-local
address of "::".

Please try setting

[addresses]
management_ip='0.0.0.0'

in your /etc/vdsm/vdsm.conf instead of the crontab hacks.


this solve the issue, but I still to haven't default gateway on 
ovirtmgmt inteface.




Using this configuration all work, but every 50 minutes I receive 3 
email from broker:


- ovirt-hosted-engine state transition StartState-ReinitializeFSM
- ovirt-hosted-engine state transition ReinitializeFSM-EngineStarting
- ovirt-hosted-engine state transition EngineStarting-EngineUp

the uptime of engine is about 15 hours, so really the engine VM is not 
rebooted.

Agent log:


MainThread::INFO::2016-06-29 
08:46:10,453::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current sta

te ReinitializeFSM (score: 0)
MainThread::INFO::2016-06-29 
08:46:20,547::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) 
Trying: notify time=1467182780.
55 type=state_transition detail=ReinitializeFSM-EngineStarting 
hostname='ovirt01.hawai.lan'
MainThread::INFO::2016-06-29 
08:46:20,732::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) 
Success, was notification of st

ate_transition (ReinitializeFSM-EngineStarting) sent? sent
MainThread::INFO::2016-06-29 
08:46:20,733::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) 
Initializin

g VDSM
MainThread::INFO::2016-06-29 
08:46:24,430::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) 
C

onnecting the storage
MainThread::INFO::2016-06-29 
08:46:24,431::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) 
Conn

ecting storage server
MainThread::INFO::2016-06-29 
08:46:31,764::storage_server::225::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) 
Conn

ecting storage server
MainThread::INFO::2016-06-29 
08:46:31,780::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) 
Refr

eshing the storage domain
MainThread::INFO::2016-06-29 
08:46:31,945::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) 
P

reparing images
MainThread::INFO::2016-06-29 
08:46:31,946::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) 
Preparing images
MainThread::INFO::2016-06-29 
08:46:35,895::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) 
R

eloading vm.conf from the shared storage domain
MainThread::INFO::2016-06-29 
08:46:35,896::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) 
Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-06-29 
08:46:39,621::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) 
Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, 
volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9
MainThread::INFO::2016-06-29 
08:46:39,667::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) 
Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, 
volUUID:3c477b06-063e-4f01-bd05-84c7d467742b
MainThread::INFO::2016-06-29 
08:46:39,760::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) 
Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-06-29 
08:46:39,761::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) 
OVF_STORE volume path: 
/rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b
MainThread::INFO::2016-06-29 
08:46:39,772::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) 
Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-06-29 
08:46:39,774::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) 
Got vm.conf from OVF_STORE
MainThread::INFO::2016-06-29 
08:46:43,489::hosted_engine::461::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineStarting (score: 3400)
MainThread::INFO::2016-06-29 
08:46:53,605::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Timeout cleared while

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-28 Thread Simone Tiraboschi 
On Tue, Jun 28, 2016 at 5:24 PM, Dan Kenigsberg  wrote:
> On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote:
>> On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg  wrote:
>> > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:
>> >> Hi!
>> >
>> >
>> > Thanks for the detailed logging!
>> >
>> >>
>> >> The broker error is:
>> >>
>> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
>> >> MainThread::INFO::2016-06-27 
>> >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>> >> Success, id 140293563619152
>> >>
>> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
>> >> Thread-25::ERROR::2016-06-27 
>> >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>> >> Error while serving connection
>> >> Traceback (most recent call last):
>> >>   File
>> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> >> line 166, in handle
>> >> data)
>> >>   File
>> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> >> line 299, in _dispatch
>> >> .set_storage_domain(client, sd_type, **options)
>> >>   File 
>> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> >> line 66, in set_storage_domain
>> >> self._backends[client].connect()
>> >>   File 
>> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> >> line 400, in connect
>> >> volUUID=volume.volume_uuid
>> >>   File 
>> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> >> line 245, in _get_volume_path
>> >> volUUID
>> >
>> > We have two issues here. First is that
>> > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
>> > is still using vdscli to contact vdsm, instead of the preferred
>> > jsonrpccli.
>>
>> We already have this one:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1101554
>>
> Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient
>
> It does not seem to be the same thing; here I'm referring to
> ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one
> condition it's the soon-to-be-deprecated xmlrpc.

ovirt-hosted-engine-setup is already using just API via jsonrpc.
The issue is that ovirt-hosted-engine-ha somewhere uses vdsClient
which AFAIK internally still uses xmlrpc
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-28 Thread Dan Kenigsberg 
On Tue, Jun 28, 2016 at 04:08:51PM +0200, Simone Tiraboschi wrote:
> On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg  wrote:
> > On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:
> >> Hi!
> >
> >
> > Thanks for the detailed logging!
> >
> >>
> >> The broker error is:
> >>
> >> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> >> MainThread::INFO::2016-06-27 
> >> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> >> Success, id 140293563619152
> >>
> >> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> >> Thread-25::ERROR::2016-06-27 
> >> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> >> Error while serving connection
> >> Traceback (most recent call last):
> >>   File
> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> >> line 166, in handle
> >> data)
> >>   File
> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> >> line 299, in _dispatch
> >> .set_storage_domain(client, sd_type, **options)
> >>   File 
> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> >> line 66, in set_storage_domain
> >> self._backends[client].connect()
> >>   File 
> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> >> line 400, in connect
> >> volUUID=volume.volume_uuid
> >>   File 
> >> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> >> line 245, in _get_volume_path
> >> volUUID
> >
> > We have two issues here. First is that
> > https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
> > is still using vdscli to contact vdsm, instead of the preferred
> > jsonrpccli.
> 
> We already have this one:
> https://bugzilla.redhat.com/show_bug.cgi?id=1101554
> 
Bug 1101554 - [RFE] HE-setup: use vdsm api instead of vdsClient

It does not seem to be the same thing; here I'm referring to
ovirt-hosted-engine-ha (not -setup). -ha *is* using vdsm API, but in one
condition it's the soon-to-be-deprecated xmlrpc.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-28 Thread Stefano Danzi 



Il 28/06/2016 15.02, Dan Kenigsberg ha scritto:

On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:

Hi!


Thanks for the detailed logging!


The broker error is:

==> /var/log/ovirt-hosted-engine-ha/agent.log <==
MainThread::INFO::2016-06-27 
09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140293563619152

==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-25::ERROR::2016-06-27 
09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Error while serving connection
Traceback (most recent call last):
   File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle
 data)
   File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch
 .set_storage_domain(client, sd_type, **options)
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain
 self._backends[client].connect()
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 400, in connect
 volUUID=volume.volume_uuid
   File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 245, in _get_volume_path
 volUUID

We have two issues here. First is that
https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
is still using vdscli to contact vdsm, instead of the preferred
jsonrpccli.

The second is that vdscli.connect's heuristic ends up reading the local
server address from vdsm config, where it finds the default ipv6-local
address of "::".

Please try setting

[addresses]
management_ip='0.0.0.0'

in your /etc/vdsm/vdsm.conf instead of the crontab hacks.


this solve the issue, but I still to haven't default gateway on 
ovirtmgmt inteface.


Would you please open a bug about the two issues
(ovirt-hosted-engine-ha and vdsm networking)?


Here: https://bugzilla.redhat.com/show_bug.cgi?id=1350883


Would you report the output of `netstats -nltp` on your host, as I do
not completely understand why no interface (not even the loopback one)
was listening on ipv6?

Here:
[root@ovirt01 ~]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address   Foreign Address State   
PID/Program name
tcp0  0 0.0.0.0:39373   0.0.0.0:* LISTEN  
2200/rpc.statd
tcp0  0 0.0.0.0:111 0.0.0.0:* LISTEN  
969/rpcbind

tcp0  0 0.0.0.0:54322   0.0.0.0:* LISTEN  943/python
tcp0  0 0.0.0.0:22  0.0.0.0:* LISTEN  1571/sshd
tcp0  0 0.0.0.0:858 0.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:49152   0.0.0.0:* LISTEN  
1929/glusterfsd
tcp0  0 0.0.0.0:49153   0.0.0.0:* LISTEN  
1968/glusterfsd
tcp0  0 0.0.0.0:20490.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:38465   0.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:38466   0.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:16514   0.0.0.0:* LISTEN  
1603/libvirtd
tcp0  0 0.0.0.0:38468   0.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:38469   0.0.0.0:* LISTEN  
1946/glusterfs
tcp0  0 0.0.0.0:24007   0.0.0.0:* LISTEN  
1585/glusterd

tcp6   0  0 :::54321:::* LISTEN  1893/python
tcp6   0  0 :::22   :::* LISTEN  1571/sshd
tcp6   0  0 :::16514:::* LISTEN  1603/libvirtd




Regards,
Dan.



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-28 Thread Simone Tiraboschi 
On Tue, Jun 28, 2016 at 3:02 PM, Dan Kenigsberg  wrote:
> On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:
>> Hi!
>
>
> Thanks for the detailed logging!
>
>>
>> The broker error is:
>>
>> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
>> MainThread::INFO::2016-06-27 
>> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>> Success, id 140293563619152
>>
>> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
>> Thread-25::ERROR::2016-06-27 
>> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>> Error while serving connection
>> Traceback (most recent call last):
>>   File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 166, in handle
>> data)
>>   File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 299, in _dispatch
>> .set_storage_domain(client, sd_type, **options)
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> line 66, in set_storage_domain
>> self._backends[client].connect()
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 400, in connect
>> volUUID=volume.volume_uuid
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 245, in _get_volume_path
>> volUUID
>
> We have two issues here. First is that
> https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
> is still using vdscli to contact vdsm, instead of the preferred
> jsonrpccli.

We already have this one:
https://bugzilla.redhat.com/show_bug.cgi?id=1101554

> The second is that vdscli.connect's heuristic ends up reading the local
> server address from vdsm config, where it finds the default ipv6-local
> address of "::".
>
> Please try setting
>
> [addresses]
> management_ip='0.0.0.0'
>
> in your /etc/vdsm/vdsm.conf instead of the crontab hacks.
>
>
> Would you please open a bug about the two issues
> (ovirt-hosted-engine-ha and vdsm networking)?
>
> Would you report the output of `netstats -nltp` on your host, as I do
> not completely understand why no interface (not even the loopback one)
> was listening on ipv6?
>
> Regards,
> Dan.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-28 Thread Dan Kenigsberg 
On Mon, Jun 27, 2016 at 10:08:33AM +0200, Stefano Danzi wrote:
> Hi!


Thanks for the detailed logging!

> 
> The broker error is:
> 
> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> MainThread::INFO::2016-06-27 
> 09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Success, id 140293563619152
> 
> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> Thread-25::ERROR::2016-06-27 
> 09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Error while serving connection
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> line 166, in handle
> data)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> line 299, in _dispatch
> .set_storage_domain(client, sd_type, **options)
>   File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> line 66, in set_storage_domain
> self._backends[client].connect()
>   File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> line 400, in connect
> volUUID=volume.volume_uuid
>   File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> line 245, in _get_volume_path
> volUUID

We have two issues here. First is that
https://gerrit.ovirt.org/gitweb?p=ovirt-hosted-engine-ha.git;a=blob;f=ovirt_hosted_engine_ha/lib/storage_backends.py;h=f2fbdc43d0e4afd7539a3a1de75de0cb07bdca9d;hb=HEAD#l271
is still using vdscli to contact vdsm, instead of the preferred
jsonrpccli.

The second is that vdscli.connect's heuristic ends up reading the local
server address from vdsm config, where it finds the default ipv6-local
address of "::".

Please try setting

[addresses]
management_ip='0.0.0.0'

in your /etc/vdsm/vdsm.conf instead of the crontab hacks.


Would you please open a bug about the two issues
(ovirt-hosted-engine-ha and vdsm networking)?

Would you report the output of `netstats -nltp` on your host, as I do
not completely understand why no interface (not even the loopback one)
was listening on ipv6?

Regards,
Dan.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-27 Thread Stefano Danzi 

Hi!

The broker error is:

==> /var/log/ovirt-hosted-engine-ha/agent.log <==
MainThread::INFO::2016-06-27 
09:27:03,311::brokerlink::140::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) 
Success, id 140293563619152


==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-25::ERROR::2016-06-27 
09:27:03,314::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) 
Error while serving connection

Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle

data)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch

.set_storage_domain(client, sd_type, **options)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain

self._backends[client].connect()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 400, in connect

volUUID=volume.volume_uuid
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 245, in _get_volume_path

volUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 797, in send
self.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, 
in connect

sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 101] Network is unreachable


Broker PID is:

[root@ovirt01 ~]# ps -ef | grep bro
vdsm   925 1  0 09:23 ?00:00:01 /usr/bin/python 
/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker --no-daemon

root  3558  3405  0 09:28 pts/000:00:00 grep --color=auto bro


[root@ovirt01 ~]# strace -f -p 925
[..]
[pid  3748] munmap(0x7f98bded, 4096) = 0
_*[pid  3748] socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 8*__*
*__*[pid  3748] connect(8, {sa_family=AF_INET6, sin6_port=htons(54321), 
inet_pton(AF_INET6, "::", _addr), sin6_flowinfo=0, 
sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)*_

[pid  3748] close(8)= 0
[pid  3748] select(0, NULL, NULL, NULL, {1, 0} 
[pid  3742] <... recvfrom resumed> "set-storage-domain VdsmBackend 
h"..., 8192, 0, NULL, NULL) = 676

[pid  3742] fcntl(6, F_GETFL)   = 0x2 (flags O_RDWR)
[pid  3742] fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid  3742] openat(AT_FDCWD, "/rhev/data-center/mnt", 
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8

[pid  3742] getdents(8, /* 7 entries */, 32768) = 312
[pid  3742] getdents(8, /* 0 entries */, 32768) = 0
[pid  3742] close(8)= 0
[pid  3742] 
access("/rhev/data-center/mnt/ovirt01.hawai.lan:_home_storage-ovirtShe/46f55a31-f35f-465c-b3e2-df45c05e06a7", 
F_OK) = -1 ENOENT (No such file or directory)
[pid  3742] 
access("/rhev/data-center/mnt/glusterSD/46f55a31-f35f-465c-b3e2-df45c05e06a7", 
F_OK) = -1 ENOENT (No such file or directory)
[pid  3742] 
access("/rhev/data-center/mnt/ovirtbk-mount.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7", 
F_OK) = -1 ENOENT (No such file or directory)
[pid  3742] 
access("/rhev/data-center/mnt/ovirtbk-sheng.hawai.lan:_var_lib_exports_iso/46f55a31-f35f-465c-b3e2-df45c05e06a7", 
F_OK) = -1 ENOENT (No such file or directory)
[pid  3742] 
access("/rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7", 
F_OK) = 0

[pid  3742] access("/etc/pki/vdsm/keys/vdsmkey.pem", R_OK) = 0
[pid  3742] access("/etc/pki/vdsm/certs/vdsmcert.pem", R_OK) = 0
[pid  3742] access("/etc/pki/vdsm/certs/cacert.pem", R_OK) = 0
[pid  3742] open("/etc/pki/vdsm/certs/vdsmcert.pem", O_RDONLY) = 8
[pid  3742] fstat(8, {st_mode=S_IFREG|0644, st_size=1574, ...}) = 0
[pid  3742] mmap(NULL, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f98bded

[pid  3742] read(8, "-BEGIN CERTIFICATE-\nMIIE"..., 4096) = 1574
[pid  3742] read(8, "", 4096)   = 0
[pid  3742] close(8)= 0
[pid  3742] munmap(0x7f98bded, 4096) = 0
[pid  3742] open("/etc/pki/vdsm/keys/vdsmkey.pem", O_RDONLY) = 8
[pid  3742] fstat(8, {st_mode=S_IFREG|0440,

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-26 Thread Dan Kenigsberg 
On Fri, Jun 24, 2016 at 06:45:24PM +0200, Stefano Danzi wrote:
> HI!!
> 
> I found a workaround
> 
> the brocker process try to connect to vdsm to IPV4 host address using an
> IPV6 connection
> (I noticed that doing a strace to the process),
> but ipv6 is not intialized at boot. (why connect to IPV4 address using
> IPV6?)

Acutally, we take an effort to disable ipv6 on ovirt host networks.
Keeping them open without explicit request was deemed a security issue.

Can you share your strace line and the relevant lines in vdsm.log? I
don't understand what is the issue that you are reporting.

> 
> I added the following lines to crontab:
> 
> @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6' |
> /usr/bin/at now+1 minutes
> @reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/ovirtmgmt/disable_ipv6' |
> /usr/bin/at now+1 minutes
> @reboot echo '/usr/sbin/route add default gw 192.168.1.254'  | /usr/bin/at
> now+1 minutes
> 
> 
> 
> Il 24/06/2016 12.36, Stefano Danzi ha scritto:
> > How I can change self hosted engine configuration to mount directly
> > gluster storage without pass through gluster NFS?
> > 
> > Maybe this solve
> > 
> > Il 24/06/2016 10.16, Stefano Danzi ha scritto:
> > > After an additional yum clean all && yum update was updated some
> > > other rpms.
> > > 
> > > Something changed.
> > > My setup has engine storage on gluster, but mounted with NFS.
> > > Now gluster daemon don't automatically start at boot. After starting
> > > manually gluster the error is the same:
> > > 
> > > ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> > > Thread-19::ERROR::2016-06-24 
> > > 10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> > > Error while serving connection
> > > Traceback (most recent call last):
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> > > line 166, in handle
> > > data)
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> > > line 299, in _dispatch
> > > .set_storage_domain(client, sd_type, **options)
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> > > line 66, in set_storage_domain
> > > self._backends[client].connect()
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> > > line 400, in connect
> > > volUUID=volume.volume_uuid
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> > > line 245, in _get_volume_path
> > > volUUID
> > >   File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
> > > return self.__send(self.__name, args)
> > >   File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
> > > verbose=self.__verbose
> > >   File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
> > > return self.single_request(host, handler, request_body, verbose)
> > >   File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
> > > self.send_content(h, request_body)
> > >   File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
> > > connection.endheaders(request_body)
> > >   File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
> > > self._send_output(message_body)
> > >   File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
> > > self.send(msg)
> > >   File "/usr/lib64/python2.7/httplib.py", line 797, in send
> > > self.connect()
> > >   File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line
> > > 203, in connect
> > > sock = socket.create_connection((self.host, self.port), self.timeout)
> > >   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
> > > raise err
> > > error: [Errno 101] Network is unreachable
> > > 
> > > 
> > > VDSM.log
> > > 
> > > jsonrpc.Executor/5::DEBUG::2016-06-24
> > > 10:10:21,694::task::995::Storage.TaskManager.Task::(_decref)
> > > Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0
> > > aborting False
> > > jsonrpc.Executor/5::WARNING::2016-06-24
> > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency)
> > > Following parameters ['type'] were not recogn
> > > ized
> > > jsonrpc.Executor/5::WARNING::2016-06-24
> > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency)
> > > Provided value "2" not defined in DiskType en
> > > um for Volume.getInfo
> > > jsonrpc.Executor/5::WARNING::2016-06-24
> > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency)
> > > Parameter capacity is not uint type
> > > jsonrpc.Executor/5::WARNING::2016-06-24
> > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency)
> > > Required property allocType is not provided w
> > > hen calling Volume.getInfo
> > > jsonrpc.Executor/5::WARNING::2016-06-24
> > > 10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency)
> > > Parameter mtime

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-24 Thread Stefano Danzi 

HI!!

I found a workaround

the brocker process try to connect to vdsm to IPV4 host address using an 
IPV6 connection

(I noticed that doing a strace to the process),
but ipv6 is not intialized at boot. (why connect to IPV4 address using 
IPV6?)


I added the following lines to crontab:

@reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6' | 
/usr/bin/at now+1 minutes
@reboot echo 'echo 0 > /proc/sys/net/ipv6/conf/ovirtmgmt/disable_ipv6' | 
/usr/bin/at now+1 minutes
@reboot echo '/usr/sbin/route add default gw 192.168.1.254'  | 
/usr/bin/at now+1 minutes




Il 24/06/2016 12.36, Stefano Danzi ha scritto:
How I can change self hosted engine configuration to mount directly 
gluster storage without pass through gluster NFS?


Maybe this solve

Il 24/06/2016 10.16, Stefano Danzi ha scritto:
After an additional yum clean all && yum update was updated some 
other rpms.


Something changed.
My setup has engine storage on gluster, but mounted with NFS.
Now gluster daemon don't automatically start at boot. After starting 
manually gluster the error is the same:


==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-19::ERROR::2016-06-24 
10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) 
Error while serving connection

Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle

data)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch

.set_storage_domain(client, sd_type, **options)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain

self._backends[client].connect()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 400, in connect

volUUID=volume.volume_uuid
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 245, in _get_volume_path

volUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 797, in send
self.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, 
in connect

sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 101] Network is unreachable


VDSM.log

jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) 
Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0

aborting False
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Following parameters ['type'] were not recogn

ized
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Provided value "2" not defined in DiskType en

um for Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter capacity is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Required property allocType is not provided w

hen calling Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter mtime is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter ctime is not int type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter truesize is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter apparentsize is not uint type
jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) 
Return 'Volume.getInfo' in bridge with {'sta
tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 
'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 
'parent': '--00
00--', 'format': 'RAW', 'image': 
'6838c974-7656-4b40-87cc-f562ff0b2a4c',

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-24 Thread Stefano Danzi 
How I can change self hosted engine configuration to mount directly 
gluster storage without pass through gluster NFS?


Maybe this solve

Il 24/06/2016 10.16, Stefano Danzi ha scritto:
After an additional yum clean all && yum update was updated some other 
rpms.


Something changed.
My setup has engine storage on gluster, but mounted with NFS.
Now gluster daemon don't automatically start at boot. After starting 
manually gluster the error is the same:


==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-19::ERROR::2016-06-24 
10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) 
Error while serving connection

Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle

data)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch

.set_storage_domain(client, sd_type, **options)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain

self._backends[client].connect()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 400, in connect

volUUID=volume.volume_uuid
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 245, in _get_volume_path

volUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 797, in send
self.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, 
in connect

sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 101] Network is unreachable


VDSM.log

jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) 
Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0

aborting False
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Following parameters ['type'] were not recogn

ized
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Provided value "2" not defined in DiskType en

um for Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter capacity is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Required property allocType is not provided w

hen calling Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter mtime is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter ctime is not int type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter truesize is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter apparentsize is not uint type
jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) 
Return 'Volume.getInfo' in bridge with {'sta
tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 
'voltype': 'LEAF', 'description': 'hosted-engine.lockspace', 'parent': 
'--00
00--', 'format': 'RAW', 'image': 
'6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 
'disktype': '2', 'legality': 'LEGAL',
'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 
'capacity': '1048576', 'uuid': 
u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 'truesize

': '1048576', 'type': 'PREALLOCATED'}
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: (104, 'Connection r

eset by peer')
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) 
unhandled close event
JsonRpc (StompReactor)::ERROR::2016-06-24

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-24 Thread Stefano Danzi 

After an additional yum clean all && yum update was updated some other rpms.

Something changed.
My setup has engine storage on gluster, but mounted with NFS.
Now gluster daemon don't automatically start at boot. After starting 
manually gluster the error is the same:


==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-19::ERROR::2016-06-24 
10:10:36,758::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) 
Error while serving connection

Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle

data)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch

.set_storage_domain(client, sd_type, **options)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain

self._backends[client].connect()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 400, in connect

volUUID=volume.volume_uuid
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", 
line 245, in _get_volume_path

volUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 797, in send
self.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, 
in connect

sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 101] Network is unreachable


VDSM.log

jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,694::task::995::Storage.TaskManager.Task::(_decref) 
Task=`5c3b6f30-d3a8-431e-9dd0-8df79b171709`::ref 0

aborting False
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Following parameters ['type'] were not recogn

ized
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Provided value "2" not defined in DiskType en

um for Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter capacity is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Required property allocType is not provided w

hen calling Volume.getInfo
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter mtime is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,694::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter ctime is not int type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter truesize is not uint type
jsonrpc.Executor/5::WARNING::2016-06-24 
10:10:21,695::vdsmapi::143::SchemaCache::(_report_inconsistency) 
Parameter apparentsize is not uint type
jsonrpc.Executor/5::DEBUG::2016-06-24 
10:10:21,695::__init__::550::jsonrpc.JsonRpcServer::(_serveRequest) 
Return 'Volume.getInfo' in bridge with {'sta
tus': 'OK', 'domain': '46f55a31-f35f-465c-b3e2-df45c05e06a7', 'voltype': 
'LEAF', 'description': 'hosted-engine.lockspace', 'parent': 
'--00
00--', 'format': 'RAW', 'image': 
'6838c974-7656-4b40-87cc-f562ff0b2a4c', 'ctime': '1423074433', 
'disktype': '2', 'legality': 'LEGAL',
'mtime': '0', 'apparentsize': '1048576', 'children': [], 'pool': '', 
'capacity': '1048576', 'uuid': u'c66a14d3-112a-4104-9025-76bb2e7ad9f1', 
'truesize

': '1048576', 'type': 'PREALLOCATED'}
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:36,514::betterAsyncore::113::vds.dispatcher::(recv) SSL error 
during reading data: (104, 'Connection r

eset by peer')
JsonRpc (StompReactor)::WARNING::2016-06-24 
10:10:36,515::betterAsyncore::154::vds.dispatcher::(log_info) unhandled 
close event
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:43,807::betterAsyncore::132::vds.dispatcher::(send) SSL error 
during sending data: (104, 'Connection r

eset by peer')
JsonRpc (StompReactor)::ERROR::2016-06-24 
10:10:43,959::betterAsyncore::113::vds.dispatcher::(recv) SSL error

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-23 Thread Stefano Danzi 

Hi!
After cleanin metadata yum do an update of vdsm:

[root@ovirt01 ~]# rpm -qva | grep vdsm
vdsm-yajsonrpc-4.18.4.1-0.el7.centos.noarch
vdsm-infra-4.18.4.1-0.el7.centos.noarch
vdsm-cli-4.18.4.1-0.el7.centos.noarch
vdsm-python-4.18.4.1-0.el7.centos.noarch
vdsm-hook-vmfex-dev-4.18.4.1-0.el7.centos.noarch
vdsm-xmlrpc-4.18.4.1-0.el7.centos.noarch
vdsm-4.18.4.1-0.el7.centos.x86_64
vdsm-api-4.18.4.1-0.el7.centos.noarch
vdsm-gluster-4.18.4.1-0.el7.centos.noarch
vdsm-jsonrpc-4.18.4.1-0.el7.centos.noarch

But this not solve the issue.

- Host haven't default gateway after a reboot
- Self Hosted engine don't start.

vdsm.log:
https://drive.google.com/file/d/0ByMG4sDqvlZcVEJ5YVI1UWxrdE0/view?usp=sharing

Il 2016-06-23 21:41 Sandro Bonazzola ha scritto:

On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi 
wrote:


Hi!
I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start
the self hosted engine.


Hi Stefano, can you please try "yum clean metadata" "yum update"
again?
You should get vdsm 4.18.4.1, please let us know if this solve your
issue.

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-23 Thread Sandro Bonazzola 
On Thu, Jun 23, 2016 at 6:36 PM, Stefano Danzi  wrote:

>
> Hi!
> I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self
> hosted engine.
>

Hi Stefano, can you please try "yum clean metadata" "yum update" again?
You should get vdsm 4.18.4.1, please let us know if this solve your issue.



>
> first thing is that the host network lose the degaut gateway
> configuration. But this is not the problem.
>
> Logs:
>
> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> MainThread::INFO::2016-06-23
> 18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2016-06-23
> 18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2016-06-23
> 18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c,
> volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9
> MainThread::INFO::2016-06-23
> 18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a,
> volUUID:3c477b06-063e-4f01-bd05-84c7d467742b
> MainThread::INFO::2016-06-23
> 18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-06-23
> 18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b
> MainThread::INFO::2016-06-23
> 18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2016-06-23
> 18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2016-06-23
> 18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
> Initializing ha-broker connection
> MainThread::INFO::2016-06-23
> 18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
> Starting monitor ping, options {'addr': '192.168.1.254'}
>
> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> Thread-25::ERROR::2016-06-23
> 18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
> Error while serving connection
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> line 166, in handle
> data)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
> line 299, in _dispatch
> .set_storage_domain(client, sd_type, **options)
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> line 66, in set_storage_domain
> self._backends[client].connect()
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> line 400, in connect
> volUUID=volume.volume_uuid
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> line 245, in _get_volume_path
> volUUID
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
> return self.__send(self.__name, args)
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
> verbose=self.__verbose
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
> return self.single_request(host, handler, request_body, verbose)
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
> self.send_content(h, request_body)
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
> connection.endheaders(request_body)
>   File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
> self._send_output(message_body)
>   File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
> self.send(msg)
>   File "/usr/lib64/python2.7/httplib.py", line 797, in send
> self.connect()
>   File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in
> connect
> sock = socket.create_connection((self.host, self.port), self.timeout)
>   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
> raise err
> error: [Errno 101] Network is unreachable
>
> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> MainThread::INFO::2016-06-23
> 18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
>

Re: [ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-23 Thread Michal Skrivanek 

> On 23 Jun 2016, at 18:36, Stefano Danzi  wrote:
> 
> 
> Hi!
> I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self 
> hosted engine.
> 
> first thing is that the host network lose the degaut gateway configuration. 
> But this is not the problem.
> 
> Logs:
> 
> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> MainThread::INFO::2016-06-23 
> 18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>  Reloading vm.conf from the shared storage domain
> MainThread::INFO::2016-06-23 
> 18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>  Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2016-06-23 
> 18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
>  Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, 
> volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9
> MainThread::INFO::2016-06-23 
> 18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
>  Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, 
> volUUID:3c477b06-063e-4f01-bd05-84c7d467742b
> MainThread::INFO::2016-06-23 
> 18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>  Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-06-23 
> 18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>  OVF_STORE volume path: 
> /rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b
> MainThread::INFO::2016-06-23 
> 18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>  Found an OVF for HE VM, trying to convert
> MainThread::INFO::2016-06-23 
> 18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>  Got vm.conf from OVF_STORE
> MainThread::INFO::2016-06-23 
> 18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
>  Initializing ha-broker connection
> MainThread::INFO::2016-06-23 
> 18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>  Starting monitor ping, options {'addr': '192.168.1.254'}
> 
> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> Thread-25::ERROR::2016-06-23 
> 18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>  Error while serving connection
> Traceback (most recent call last):
>  File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
> line 166, in handle
>data)
>  File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
> line 299, in _dispatch
>.set_storage_domain(client, sd_type, **options)
>  File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>  line 66, in set_storage_domain
>self._backends[client].connect()
>  File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>  line 400, in connect
>volUUID=volume.volume_uuid
>  File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>  line 245, in _get_volume_path
>volUUID
>  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
>return self.__send(self.__name, args)
>  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
>verbose=self.__verbose
>  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
>return self.single_request(host, handler, request_body, verbose)
>  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
>self.send_content(h, request_body)
>  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
>connection.endheaders(request_body)
>  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
>self._send_output(message_body)
>  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
>self.send(msg)
>  File "/usr/lib64/python2.7/httplib.py", line 797, in send
>self.connect()
>  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in 
> connect
>sock = socket.create_connection((self.host, self.port), self.timeout)
>  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
>raise err
> error: [Errno 101] Network is unreachable
> 
> ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> MainThread::INFO::2016-06-23 
> 18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
>  Failed set the storage domain: 'Failed to set storage domain VdsmBackend, 
> options {'hosted-engine.lockspace': 
>

[ovirt-users] Failed to start self hosted engine after upgrading oVirt to 4.0

2016-06-23 Thread Stefano Danzi 


Hi!
I've just upgrade oVirt from 3.6 to 4.0 and I'm not able to start the self 
hosted engine.

first thing is that the host network lose the degaut gateway configuration. But 
this is not the problem.

Logs:

==> /var/log/ovirt-hosted-engine-ha/agent.log <==
MainThread::INFO::2016-06-23 
18:28:40,833::hosted_engine::669::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
 Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-06-23 
18:28:40,833::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
 Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-06-23 
18:28:44,535::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
 Found OVF_STORE: imgUUID:8d07965c-a5c4-4057-912d-901f80cf246c, 
volUUID:ce3aa63e-e1c4-498e-bdca-9d2e9f47f0f9
MainThread::INFO::2016-06-23 
18:28:44,582::ovf_store::102::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
 Found OVF_STORE: imgUUID:bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a, 
volUUID:3c477b06-063e-4f01-bd05-84c7d467742b
MainThread::INFO::2016-06-23 
18:28:44,674::ovf_store::111::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
 Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-06-23 
18:28:44,675::ovf_store::118::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
 OVF_STORE volume path: 
/rhev/data-center/mnt/ovirt01.hawai.lan:_engine/46f55a31-f35f-465c-b3e2-df45c05e06a7/images/bd9aaf0b-8435-4d78-9871-8c7a7f7fa02a/3c477b06-063e-4f01-bd05-84c7d467742b
MainThread::INFO::2016-06-23 
18:28:44,682::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
 Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-06-23 
18:28:44,684::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
 Got vm.conf from OVF_STORE
MainThread::INFO::2016-06-23 
18:28:44,684::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Initializing ha-broker connection
MainThread::INFO::2016-06-23 
18:28:44,685::brokerlink::129::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor ping, options {'addr': '192.168.1.254'}

==> /var/log/ovirt-hosted-engine-ha/broker.log <==
Thread-25::ERROR::2016-06-23 
18:28:44,697::listener::182::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
 Error while serving connection
Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle
data)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch
.set_storage_domain(client, sd_type, **options)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
 line 66, in set_storage_domain
self._backends[client].connect()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
 line 400, in connect
volUUID=volume.volume_uuid
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
 line 245, in _get_volume_path
volUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request
self.send_content(h, request_body)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content
connection.endheaders(request_body)
  File "/usr/lib64/python2.7/httplib.py", line 975, in endheaders
self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 835, in _send_output
self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 797, in send
self.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/m2cutils.py", line 203, in connect
sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 101] Network is unreachable

==> /var/log/ovirt-hosted-engine-ha/agent.log <==
MainThread::INFO::2016-06-23 
18:28:44,697::hosted_engine::602::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Failed set the storage domain: 'Failed to set storage domain VdsmBackend, options 
{'hosted-engine.lockspace': 
'7B22696D6167655F75756964223A202265663131373139322D623564662D346534362D383939622D6262663862663135222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A202230613363393433652D633032392D343134372D623364342D396366353364663161356262227D',
 'sp_uuid':