Re: [Users] hosted engine help

2014-03-25 Thread Jiri Moskovcak

On 03/13/2014 05:25 PM, Jason Brooks wrote:



- Original Message -

From: Greg Padgett gpadg...@redhat.com
To: Jason Brooks jbro...@redhat.com
Cc: Sandro Bonazzola sbona...@redhat.com, users@ovirt.org, Martin Sivak 
msi...@redhat.com
Sent: Tuesday, March 11, 2014 7:52:42 AM
Subject: Re: [Users] hosted engine help

On 03/11/2014 04:09 PM, Sandro Bonazzola wrote:

Il 07/03/2014 01:10, Jason Brooks ha scritto:

Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
while I've managed to bring the engine up, I've only been able to do it
manually, using hosted-engine --vm-start.

The ovirt-ha-agent service fails reliably for me, erroring out with
RequestError: Request failed: success.

I've pasted error passages from the ha agent and vdsm logs below.

Any pointers?

Regards, Jason

***

ovirt-ha-agent.log

MainThread::CRITICAL::2014-03-06
18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
Could not start ha-agent
Traceback (most recent call last):
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
line 97, in run
  self._run_agent()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
line 154, in _run_agent

hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
line 303, in start_monitoring
  for old_state, state, delay in self.fsm:
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py,
line 125, in next
  new_data = self.refresh(self._state.data)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
line 77, in refresh
  stats.update(self.hosted_engine.collect_stats())
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
line 623, in collect_stats
  constants.SERVICE_TYPE)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 171, in get_stats_from_storage
  result = self._checked_communicate(request)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 198, in _checked_communicate
  raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success


vdsm.log

Thread-29::ERROR::2014-03-06 18:48:11,101::API::1607::vds::(_getHaInfo)
failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
  stats = instance.get_all_stats()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py,
line 86, in get_all_stats
  constants.SERVICE_TYPE)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 171, in get_stats_from_storage
  result = self._checked_communicate(request)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 198, in _checked_communicate
  raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success



Greg, Martin, Request failed: success ?


Hi Jason,

I talked to Martin about this and opened a bug [1]/submitted a patch [2].
   Based on your mail, I'm not sure if you experienced a race condition or
some other issue.  This patch should help the former case, but if you're
still experiencing problems then we would need to investigate further.


I made these changes to my install and now I get a different error:

MainThread::CRITICAL::2014-03-13 
12:05:47,749::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could 
not start ha-agent
Traceback (most recent call last):
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, line 
97, in run
 self._run_agent()
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, line 
154, in _run_agent
 hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
 line 303, in start_monitoring
 for old_state, state, delay in self.fsm:
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py, 
line 125, in next
 new_data = self.refresh(self._state.data)
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
 line 77, in refresh
 stats.update(self.hosted_engine.collect_stats())
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
 line 623, in collect_stats
 constants.SERVICE_TYPE)
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File 
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 191, in _checked_communicate
 return parts[1]
IndexError: list index out of range

I'm attaching my vdsm.log, agent.log

Re: [Users] hosted engine help

2014-03-25 Thread Jason Brooks


- Original Message -
 From: Jiri Moskovcak jmosk...@redhat.com
 To: Jason Brooks jbro...@redhat.com, Greg Padgett gpadg...@redhat.com
 Cc: users@ovirt.org
 Sent: Tuesday, March 25, 2014 10:34:16 AM
 Subject: Re: [Users] hosted engine help
 
 On 03/13/2014 05:25 PM, Jason Brooks wrote:
 
 
  - Original Message -
  From: Greg Padgett gpadg...@redhat.com
  To: Jason Brooks jbro...@redhat.com
  Cc: Sandro Bonazzola sbona...@redhat.com, users@ovirt.org, Martin
  Sivak msi...@redhat.com
  Sent: Tuesday, March 11, 2014 7:52:42 AM
  Subject: Re: [Users] hosted engine help
 
  On 03/11/2014 04:09 PM, Sandro Bonazzola wrote:
  Il 07/03/2014 01:10, Jason Brooks ha scritto:
  Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
  while I've managed to bring the engine up, I've only been able to do it
  manually, using hosted-engine --vm-start.
 
  The ovirt-ha-agent service fails reliably for me, erroring out with
  RequestError: Request failed: success.
 
  I've pasted error passages from the ha agent and vdsm logs below.
 
  Any pointers?
 
  Regards, Jason
 
  ***
 
  ovirt-ha-agent.log
 
  MainThread::CRITICAL::2014-03-06
  18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
  Could not start ha-agent
  Traceback (most recent call last):
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
  line 97, in run
self._run_agent()
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
  line 154, in _run_agent
 
  hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 303, in start_monitoring
for old_state, state, delay in self.fsm:
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py,
  line 125, in next
new_data = self.refresh(self._state.data)
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
  line 77, in refresh
stats.update(self.hosted_engine.collect_stats())
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 623, in collect_stats
constants.SERVICE_TYPE)
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
  line 171, in get_stats_from_storage
result = self._checked_communicate(request)
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
  line 198, in _checked_communicate
raise RequestError(Request failed: {0}.format(msg))
  RequestError: Request failed: success
 
 
  vdsm.log
 
  Thread-29::ERROR::2014-03-06 18:48:11,101::API::1607::vds::(_getHaInfo)
  failed to retrieve Hosted Engine HA info
  Traceback (most recent call last):
  File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
stats = instance.get_all_stats()
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py,
  line 86, in get_all_stats
constants.SERVICE_TYPE)
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
  line 171, in get_stats_from_storage
result = self._checked_communicate(request)
  File
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
  line 198, in _checked_communicate
raise RequestError(Request failed: {0}.format(msg))
  RequestError: Request failed: success
 
 
  Greg, Martin, Request failed: success ?
 
  Hi Jason,
 
  I talked to Martin about this and opened a bug [1]/submitted a patch [2].
 Based on your mail, I'm not sure if you experienced a race condition or
  some other issue.  This patch should help the former case, but if you're
  still experiencing problems then we would need to investigate further.
 
  I made these changes to my install and now I get a different error:
 
  MainThread::CRITICAL::2014-03-13
  12:05:47,749::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
  Could not start ha-agent
  Traceback (most recent call last):
 File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
 line 97, in run
   self._run_agent()
 File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
 line 154, in _run_agent
   hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
 File
 
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
 line 303, in start_monitoring
   for old_state, state, delay in self.fsm:
 File
 
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py,
 line 125, in next
   new_data = self.refresh(self._state.data)
 File
 
  /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
 line 77, in refresh
   stats.update(self.hosted_engine.collect_stats())
 File
 
  /usr/lib

Re: [Users] hosted engine help

2014-03-25 Thread Jiri Moskovcak

On 03/25/2014 06:34 PM, Jiri Moskovcak wrote:

On 03/13/2014 05:25 PM, Jason Brooks wrote:



- Original Message -

From: Greg Padgett gpadg...@redhat.com
To: Jason Brooks jbro...@redhat.com
Cc: Sandro Bonazzola sbona...@redhat.com, users@ovirt.org,
Martin Sivak msi...@redhat.com
Sent: Tuesday, March 11, 2014 7:52:42 AM
Subject: Re: [Users] hosted engine help

On 03/11/2014 04:09 PM, Sandro Bonazzola wrote:

Il 07/03/2014 01:10, Jason Brooks ha scritto:

Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
while I've managed to bring the engine up, I've only been able to
do it
manually, using hosted-engine --vm-start.

The ovirt-ha-agent service fails reliably for me, erroring out with
RequestError: Request failed: success.

I've pasted error passages from the ha agent and vdsm logs below.

Any pointers?

Regards, Jason

***

ovirt-ha-agent.log

MainThread::CRITICAL::2014-03-06
18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)

Could not start ha-agent
Traceback (most recent call last):
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,

line 97, in run
  self._run_agent()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,

line 154, in _run_agent

hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,

line 303, in start_monitoring
  for old_state, state, delay in self.fsm:
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py,

line 125, in next
  new_data = self.refresh(self._state.data)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,

line 77, in refresh
  stats.update(self.hosted_engine.collect_stats())
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,

line 623, in collect_stats
  constants.SERVICE_TYPE)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,

line 171, in get_stats_from_storage
  result = self._checked_communicate(request)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,

line 198, in _checked_communicate
  raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success


vdsm.log

Thread-29::ERROR::2014-03-06
18:48:11,101::API::1607::vds::(_getHaInfo)
failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
  stats = instance.get_all_stats()
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py,

line 86, in get_all_stats
  constants.SERVICE_TYPE)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,

line 171, in get_stats_from_storage
  result = self._checked_communicate(request)
File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,

line 198, in _checked_communicate
  raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success



Greg, Martin, Request failed: success ?


Hi Jason,

I talked to Martin about this and opened a bug [1]/submitted a patch
[2].
   Based on your mail, I'm not sure if you experienced a race
condition or
some other issue.  This patch should help the former case, but if you're
still experiencing problems then we would need to investigate further.


I made these changes to my install and now I get a different error:

MainThread::CRITICAL::2014-03-13
12:05:47,749::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
Could not start ha-agent
Traceback (most recent call last):
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
line 97, in run
 self._run_agent()
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py,
line 154, in _run_agent

hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
line 303, in start_monitoring
 for old_state, state, delay in self.fsm:
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py,
line 125, in next
 new_data = self.refresh(self._state.data)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
line 77, in refresh
 stats.update(self.hosted_engine.collect_stats())
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
line 623, in collect_stats
 constants.SERVICE_TYPE)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py,
line 191, in _checked_communicate
 return parts[1]
IndexError: list index out of range

I'm

Re: [Users] hosted engine help

2014-03-11 Thread Sandro Bonazzola
Il 07/03/2014 01:10, Jason Brooks ha scritto:
 Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
 while I've managed to bring the engine up, I've only been able to do it
 manually, using hosted-engine --vm-start.
 
 The ovirt-ha-agent service fails reliably for me, erroring out with
 RequestError: Request failed: success.
 
 I've pasted error passages from the ha agent and vdsm logs below.
 
 Any pointers?
 
 Regards, Jason
 
 ***
 
 ovirt-ha-agent.log
 
 MainThread::CRITICAL::2014-03-06
 18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) 
 Could not start ha-agent
 Traceback (most recent call last):
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, 
 line 97, in run
 self._run_agent()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, 
 line 154, in _run_agent
 
 hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 303, in start_monitoring
 for old_state, state, delay in self.fsm:
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py, 
 line 125, in next
 new_data = self.refresh(self._state.data)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
  line 77, in refresh
 stats.update(self.hosted_engine.collect_stats())
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 623, in collect_stats
 constants.SERVICE_TYPE)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
 RequestError: Request failed: success
 
 
 vdsm.log
 
 Thread-29::ERROR::2014-03-06 18:48:11,101::API::1607::vds::(_getHaInfo)
 failed to retrieve Hosted Engine HA info
 Traceback (most recent call last):
   File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
 stats = instance.get_all_stats()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py, 
 line 86, in get_all_stats
 constants.SERVICE_TYPE)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
 RequestError: Request failed: success


Greg, Martin, Request failed: success ?

 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-11 Thread Sandro Bonazzola
Il 10/03/2014 22:32, Giuseppe Ragusa ha scritto:
 Hi all,
 
 Date: Mon, 10 Mar 2014 12:56:19 -0400
 From: jbro...@redhat.com
 To: msi...@redhat.com
 CC: users@ovirt.org
 Subject: Re: [Users] hosted engine help



 - Original Message -
  From: Martin Sivak msi...@redhat.com
  To: Dan Kenigsberg dan...@redhat.com
  Cc: users@ovirt.org
  Sent: Saturday, March 8, 2014 11:52:59 PM
  Subject: Re: [Users] hosted engine help
 
  Hi Jason,
 
  can you please attach the full logs? We had very similar issue before I we
  need to see if is the same or not.

 I may have to recreate it -- I switched back to an all in one engine after my
 setup started refusing to run the engine at all. It's no fun losing your 
 engine!

 This was a migrated-from-standalone setup, maybe that caused additional 
 wrinkles...

 Jason

 
  Thanks
 
 I experienced the exact same symptoms as Jason on a from-scratch installation 
 on two physical nodes with CentOS 6.5 (fully up-to-date) using oVirt
 3.4.0_pre (latest test-day release) and GlusterFS 3.5.0beta3 (with 
 Gluster-provided NFS as storage for the self-hosted engine VM only).

Using GlusterFS with hosted-engine storage is not supported and not recommended.
HA daemon may not work properly there.

 
 I roughly followed the guide from Andrew Lau:
 
 http://www.andrewklau.com/ovirt-hosted-engine-with-3-4-0-nightly/
 
 with some variations due to newer packages (resolved bugs) and different 
 hardware setup (no VLANs in my setup: physically separated networks; custom
 second nic added to Engine VM template before deploying etc.)
 
 The self-hosted installation on first node + Engine VM (configured for 
 managing both oVirt and the storage; Datacenter default set to NFS because no
 GlusterFS offered) went apparently smooth, but the HA-agent failed to start 
 at the very end (same errors in logs as Jason: the storage domain seems
 missing) and I was only able to start it all manually with:
 
 hosted-engine --connect-storage
 hosted-engine --start-pool

The above commands are used for development and shouldn't be used for starting 
the engine.

 hosted-engine --vm-start
 
 then the Engine came up and I could use it, I even registered the second node 
 (same final error in HA-agent) and tried to add GlusterFS storage
 domains for further VMs and ISOs (by the way: the original NFS-GlusterFS 
 domain for Engine VM only is not present inside the Engine web UI) but it
 always failed activating the domains (they remain Inactive).
 
 Furthermore the engine gets killed some time after starting (from 3 up to 11 
 hours later) and the only way to get it back is repeating the above commands.

Need logs for this.


 
 I always managed GlusterFS natively (not through oVirt) from the 
 commandline and verified that the NFS-exported Engine-VM-only volume gets
 replicated, but I obviously failed to try migration because the HA part 
 results inactive and oVirt refuse to migrate the Engine.
 
 Since I tried many times, with variations and further manual actions between 
 (like trying to manually mount the NFS Engine domain, restarting the
 HA-agent only etc.), my logs are cluttered, so I should start from scratch 
 again and pack up all logs in one swipe.

+1

 
 Tell me what I should capture and at which points in the whole process and I 
 will try to follow up as soon as possible.

What:
hosted-engine-setup, hosted-engine-ha, vdsm, libvirt, sanlock from the physical 
hosts and engine and server logs from the hosted engine VM.

When:
As soon as you see an error.



 
 Many thanks,
 Giuseppe
 
  --
  Martin Sivák
  msi...@redhat.com
  Red Hat Czech
  RHEV-M SLA / Brno, CZ
 
  - Original Message -
   On Fri, Mar 07, 2014 at 10:17:43AM +0100, Sandro Bonazzola wrote:
Il 07/03/2014 01:10, Jason Brooks ha scritto:
 Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
 while I've managed to bring the engine up, I've only been able to do 
 it
 manually, using hosted-engine --vm-start.

 The ovirt-ha-agent service fails reliably for me, erroring out with
 RequestError: Request failed: success.

 I've pasted error passages from the ha agent and vdsm logs below.

 Any pointers?
   
looks like a VDSM bug, Dan?
  
   Why? The exception is raised from deep inside the ovirt_hosted_engine_ha
   code.
   ___
   Users mailing list
   Users@ovirt.org
   http://lists.ovirt.org/mailman/listinfo/users
  
  ___
  Users mailing list
  Users@ovirt.org
  http://lists.ovirt.org/mailman/listinfo/users
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration

Re: [Users] hosted engine help

2014-03-11 Thread Jason Brooks

  
  I experienced the exact same symptoms as Jason on a from-scratch
  installation on two physical nodes with CentOS 6.5 (fully up-to-date)
  using oVirt
  3.4.0_pre (latest test-day release) and GlusterFS 3.5.0beta3 (with
  Gluster-provided NFS as storage for the self-hosted engine VM only).
 
 Using GlusterFS with hosted-engine storage is not supported and not
 recommended.
 HA daemon may not work properly there.

I used Gluster NFS, too. That's not supported or recommended?

I've assumed that I could use Gluster NFS / NFS interchangeably...

Jason

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-11 Thread Itamar Heim

On 03/11/2014 04:41 PM, Jason Brooks wrote:




I experienced the exact same symptoms as Jason on a from-scratch
installation on two physical nodes with CentOS 6.5 (fully up-to-date)
using oVirt
3.4.0_pre (latest test-day release) and GlusterFS 3.5.0beta3 (with
Gluster-provided NFS as storage for the self-hosted engine VM only).


Using GlusterFS with hosted-engine storage is not supported and not
recommended.
HA daemon may not work properly there.


I used Gluster NFS, too. That's not supported or recommended?

I've assumed that I could use Gluster NFS / NFS interchangeably...


that should be ok. make sure your gluster environment is robust with 
quorum, etc.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-11 Thread Greg Padgett

On 03/11/2014 04:09 PM, Sandro Bonazzola wrote:

Il 07/03/2014 01:10, Jason Brooks ha scritto:

Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
while I've managed to bring the engine up, I've only been able to do it
manually, using hosted-engine --vm-start.

The ovirt-ha-agent service fails reliably for me, erroring out with
RequestError: Request failed: success.

I've pasted error passages from the ha agent and vdsm logs below.

Any pointers?

Regards, Jason

***

ovirt-ha-agent.log

MainThread::CRITICAL::2014-03-06
18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could 
not start ha-agent
Traceback (most recent call last):
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, line 
97, in run
 self._run_agent()
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, line 
154, in _run_agent

hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
 line 303, in start_monitoring
 for old_state, state, delay in self.fsm:
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py, 
line 125, in next
 new_data = self.refresh(self._state.data)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
 line 77, in refresh
 stats.update(self.hosted_engine.collect_stats())
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
 line 623, in collect_stats
 constants.SERVICE_TYPE)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success


vdsm.log

Thread-29::ERROR::2014-03-06 18:48:11,101::API::1607::vds::(_getHaInfo)
failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
   File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
 stats = instance.get_all_stats()
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py, 
line 86, in get_all_stats
 constants.SERVICE_TYPE)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
RequestError: Request failed: success



Greg, Martin, Request failed: success ?


Hi Jason,

I talked to Martin about this and opened a bug [1]/submitted a patch [2]. 
 Based on your mail, I'm not sure if you experienced a race condition or 
some other issue.  This patch should help the former case, but if you're 
still experiencing problems then we would need to investigate further.


Thanks,
Greg

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1075126
[2] http://gerrit.ovirt.org/#/c/25621/




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users






___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-11 Thread Giuseppe Ragusa
 Date: Tue, 11 Mar 2014 15:16:36 +0100
 From: sbona...@redhat.com
 To: giuseppe.rag...@hotmail.com; jbro...@redhat.com; msi...@redhat.com
 CC: users@ovirt.org; fsimo...@redhat.com; gpadg...@redhat.com
 Subject: Re: [Users] hosted engine help
 
 Il 10/03/2014 22:32, Giuseppe Ragusa ha scritto:
  Hi all,
  
  Date: Mon, 10 Mar 2014 12:56:19 -0400
  From: jbro...@redhat.com
  To: msi...@redhat.com
  CC: users@ovirt.org
  Subject: Re: [Users] hosted engine help
 
 
 
  - Original Message -
   From: Martin Sivak msi...@redhat.com
   To: Dan Kenigsberg dan...@redhat.com
   Cc: users@ovirt.org
   Sent: Saturday, March 8, 2014 11:52:59 PM
   Subject: Re: [Users] hosted engine help
  
   Hi Jason,
  
   can you please attach the full logs? We had very similar issue before I 
   we
   need to see if is the same or not.
 
  I may have to recreate it -- I switched back to an all in one engine after 
  my
  setup started refusing to run the engine at all. It's no fun losing your 
  engine!
 
  This was a migrated-from-standalone setup, maybe that caused additional 
  wrinkles...
 
  Jason
 
  
   Thanks
  
  I experienced the exact same symptoms as Jason on a from-scratch 
  installation on two physical nodes with CentOS 6.5 (fully up-to-date) using 
  oVirt
  3.4.0_pre (latest test-day release) and GlusterFS 3.5.0beta3 (with 
  Gluster-provided NFS as storage for the self-hosted engine VM only).
 
 Using GlusterFS with hosted-engine storage is not supported and not 
 recommended.
 HA daemon may not work properly there.

If it is unsupported (and particularly not recommended) even with the 
interposed NFS (the native Gluster-provided NFSv3 export of a volume), then 
which is the recommended way to setup a fault-tolerant load-balanced 2 node 
oVirt cluster (without external dedicated SAN/NAS)?

  I roughly followed the guide from Andrew Lau:
  
  http://www.andrewklau.com/ovirt-hosted-engine-with-3-4-0-nightly/
  
  with some variations due to newer packages (resolved bugs) and different 
  hardware setup (no VLANs in my setup: physically separated networks; custom
  second nic added to Engine VM template before deploying etc.)
  
  The self-hosted installation on first node + Engine VM (configured for 
  managing both oVirt and the storage; Datacenter default set to NFS because 
  no
  GlusterFS offered) went apparently smooth, but the HA-agent failed to start 
  at the very end (same errors in logs as Jason: the storage domain seems
  missing) and I was only able to start it all manually with:
  
  hosted-engine --connect-storage
  hosted-engine --start-pool
 
 The above commands are used for development and shouldn't be used for 
 starting the engine.

Directly starting the engine (with the command below) failed because of storage 
unavailability, so I used the above trick as a last resort to at least 
prove that the engine was able to start and had not been somewhat destroyed 
or lost in the process (but I do understand that it is an extreme debug-only 
action).

  hosted-engine --vm-start
  
  then the Engine came up and I could use it, I even registered the second 
  node (same final error in HA-agent) and tried to add GlusterFS storage
  domains for further VMs and ISOs (by the way: the original NFS-GlusterFS 
  domain for Engine VM only is not present inside the Engine web UI) but it
  always failed activating the domains (they remain Inactive).
  
  Furthermore the engine gets killed some time after starting (from 3 up to 
  11 hours later) and the only way to get it back is repeating the above 
  commands.
 
 Need logs for this.

I will try to reproduce it all, but I can recall that on libvirt logs 
(HostedEngine.log) there was always clear indication of the PID that killed the 
Engine VM and each time it belonged to an instance of sanlock.

  I always managed GlusterFS natively (not through oVirt) from the 
  commandline and verified that the NFS-exported Engine-VM-only volume gets
  replicated, but I obviously failed to try migration because the HA part 
  results inactive and oVirt refuse to migrate the Engine.
  
  Since I tried many times, with variations and further manual actions 
  between (like trying to manually mount the NFS Engine domain, restarting the
  HA-agent only etc.), my logs are cluttered, so I should start from 
  scratch again and pack up all logs in one swipe.
 
 +1

;

  Tell me what I should capture and at which points in the whole process and 
  I will try to follow up as soon as possible.
 
 What:
 hosted-engine-setup, hosted-engine-ha, vdsm, libvirt, sanlock from the 
 physical hosts and engine and server logs from the hosted engine VM.
 
 When:
 As soon as you see an error.

If the setup design (wholly GlusterFS based) is somewhat flawed, please point 
me to some hint/docs/guide for the right way of setting it up on 2 standalone 
physical nodes, so as not to waste your time in chasing defects in something 
that is not supposed to be working anyway.

I will follow your

Re: [Users] hosted engine help

2014-03-10 Thread Jason Brooks


- Original Message -
 From: Martin Sivak msi...@redhat.com
 To: Dan Kenigsberg dan...@redhat.com
 Cc: users@ovirt.org
 Sent: Saturday, March 8, 2014 11:52:59 PM
 Subject: Re: [Users] hosted engine help
 
 Hi Jason,
 
 can you please attach the full logs? We had very similar issue before I we
 need to see if is the same or not.

I may have to recreate it -- I switched back to an all in one engine after my
setup started refusing to run the engine at all. It's no fun losing your engine!

This was a migrated-from-standalone setup, maybe that caused additional 
wrinkles...

Jason

 
 Thanks
 
 --
 Martin Sivák
 msi...@redhat.com
 Red Hat Czech
 RHEV-M SLA / Brno, CZ
 
 - Original Message -
  On Fri, Mar 07, 2014 at 10:17:43AM +0100, Sandro Bonazzola wrote:
   Il 07/03/2014 01:10, Jason Brooks ha scritto:
Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
while I've managed to bring the engine up, I've only been able to do it
manually, using hosted-engine --vm-start.

The ovirt-ha-agent service fails reliably for me, erroring out with
RequestError: Request failed: success.

I've pasted error passages from the ha agent and vdsm logs below.

Any pointers?
   
   looks like a VDSM bug, Dan?
  
  Why? The exception is raised from deep inside the ovirt_hosted_engine_ha
  code.
  ___
  Users mailing list
  Users@ovirt.org
  http://lists.ovirt.org/mailman/listinfo/users
  
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-10 Thread Giuseppe Ragusa
Hi all,
 Date: Mon, 10 Mar 2014 12:56:19 -0400
 From: jbro...@redhat.com
 To: msi...@redhat.com
 CC: users@ovirt.org
 Subject: Re: [Users] hosted engine help
 
 
 
 - Original Message -
  From: Martin Sivak msi...@redhat.com
  To: Dan Kenigsberg dan...@redhat.com
  Cc: users@ovirt.org
  Sent: Saturday, March 8, 2014 11:52:59 PM
  Subject: Re: [Users] hosted engine help
  
  Hi Jason,
  
  can you please attach the full logs? We had very similar issue before I we
  need to see if is the same or not.
 
 I may have to recreate it -- I switched back to an all in one engine after my
 setup started refusing to run the engine at all. It's no fun losing your 
 engine!
 
 This was a migrated-from-standalone setup, maybe that caused additional 
 wrinkles...
 
 Jason
 
  
  Thanks

I experienced the exact same symptoms as Jason on a from-scratch installation 
on two physical nodes with CentOS 6.5 (fully up-to-date) using oVirt 3.4.0_pre 
(latest test-day release) and GlusterFS 3.5.0beta3 (with Gluster-provided NFS 
as storage for the self-hosted engine VM only).
I roughly followed the guide from Andrew Lau:
http://www.andrewklau.com/ovirt-hosted-engine-with-3-4-0-nightly/
with some variations due to newer packages (resolved bugs) and different 
hardware setup (no VLANs in my setup: physically separated networks; custom 
second nic added to Engine VM template before deploying etc.)
The self-hosted installation on first node + Engine VM (configured for managing 
both oVirt and the storage; Datacenter default set to NFS because no GlusterFS 
offered) went apparently smooth, but the HA-agent failed to start at the very 
end (same errors in logs as Jason: the storage domain seems missing) and I 
was only able to start it all manually with:
hosted-engine --connect-storagehosted-engine --start-poolhosted-engine 
--vm-start
then the Engine came up and I could use it, I even registered the second node 
(same final error in HA-agent) and tried to add GlusterFS storage domains for 
further VMs and ISOs (by the way: the original NFS-GlusterFS domain for Engine 
VM only is not present inside the Engine web UI) but it always failed 
activating the domains (they remain Inactive).
Furthermore the engine gets killed some time after starting (from 3 up to 11 
hours later) and the only way to get it back is repeating the above commands.
I always managed GlusterFS natively (not through oVirt) from the commandline 
and verified that the NFS-exported Engine-VM-only volume gets replicated, but I 
obviously failed to try migration because the HA part results inactive and 
oVirt refuse to migrate the Engine.
Since I tried many times, with variations and further manual actions between 
(like trying to manually mount the NFS Engine domain, restarting the HA-agent 
only etc.), my logs are cluttered, so I should start from scratch again and 
pack up all logs in one swipe.
Tell me what I should capture and at which points in the whole process and I 
will try to follow up as soon as possible.
Many thanks,Giuseppe
  --
  Martin Sivák
  msi...@redhat.com
  Red Hat Czech
  RHEV-M SLA / Brno, CZ
  
  - Original Message -
   On Fri, Mar 07, 2014 at 10:17:43AM +0100, Sandro Bonazzola wrote:
Il 07/03/2014 01:10, Jason Brooks ha scritto:
 Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
 while I've managed to bring the engine up, I've only been able to do 
 it
 manually, using hosted-engine --vm-start.
 
 The ovirt-ha-agent service fails reliably for me, erroring out with
 RequestError: Request failed: success.
 
 I've pasted error passages from the ha agent and vdsm logs below.
 
 Any pointers?

looks like a VDSM bug, Dan?
   
   Why? The exception is raised from deep inside the ovirt_hosted_engine_ha
   code.
   ___
   Users mailing list
   Users@ovirt.org
   http://lists.ovirt.org/mailman/listinfo/users
   
  ___
  Users mailing list
  Users@ovirt.org
  http://lists.ovirt.org/mailman/listinfo/users
  
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
  ___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-08 Thread Martin Sivak
Hi Jason,

can you please attach the full logs? We had very similar issue before I we need 
to see if is the same or not.

Thanks

--
Martin Sivák
msi...@redhat.com
Red Hat Czech
RHEV-M SLA / Brno, CZ

- Original Message -
 On Fri, Mar 07, 2014 at 10:17:43AM +0100, Sandro Bonazzola wrote:
  Il 07/03/2014 01:10, Jason Brooks ha scritto:
   Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
   while I've managed to bring the engine up, I've only been able to do it
   manually, using hosted-engine --vm-start.
   
   The ovirt-ha-agent service fails reliably for me, erroring out with
   RequestError: Request failed: success.
   
   I've pasted error passages from the ha agent and vdsm logs below.
   
   Any pointers?
  
  looks like a VDSM bug, Dan?
 
 Why? The exception is raised from deep inside the ovirt_hosted_engine_ha
 code.
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-07 Thread Sandro Bonazzola
Il 07/03/2014 01:10, Jason Brooks ha scritto:
 Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
 while I've managed to bring the engine up, I've only been able to do it
 manually, using hosted-engine --vm-start.
 
 The ovirt-ha-agent service fails reliably for me, erroring out with
 RequestError: Request failed: success.
 
 I've pasted error passages from the ha agent and vdsm logs below.
 
 Any pointers?

looks like a VDSM bug, Dan?

 
 Regards, Jason
 
 ***
 
 ovirt-ha-agent.log
 
 MainThread::CRITICAL::2014-03-06
 18:48:30,622::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) 
 Could not start ha-agent
 Traceback (most recent call last):
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, 
 line 97, in run
 self._run_agent()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py, 
 line 154, in _run_agent
 
 hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 303, in start_monitoring
 for old_state, state, delay in self.fsm:
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py, 
 line 125, in next
 new_data = self.refresh(self._state.data)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py,
  line 77, in refresh
 stats.update(self.hosted_engine.collect_stats())
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py,
  line 623, in collect_stats
 constants.SERVICE_TYPE)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
 RequestError: Request failed: success
 
 
 vdsm.log
 
 Thread-29::ERROR::2014-03-06 18:48:11,101::API::1607::vds::(_getHaInfo)
 failed to retrieve Hosted Engine HA info
 Traceback (most recent call last):
   File /usr/share/vdsm/API.py, line 1598, in _getHaInfo
 stats = instance.get_all_stats()
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py, 
 line 86, in get_all_stats
 constants.SERVICE_TYPE)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 171, in get_stats_from_storage
 result = self._checked_communicate(request)
   File
 /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py, 
 line 198, in _checked_communicate
 raise RequestError(Request failed: {0}.format(msg))
 RequestError: Request failed: success
 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [Users] hosted engine help

2014-03-07 Thread Dan Kenigsberg
On Fri, Mar 07, 2014 at 10:17:43AM +0100, Sandro Bonazzola wrote:
 Il 07/03/2014 01:10, Jason Brooks ha scritto:
  Hey everyone, I've been testing out oVirt 3.4 w/ hosted engine, and
  while I've managed to bring the engine up, I've only been able to do it
  manually, using hosted-engine --vm-start.
  
  The ovirt-ha-agent service fails reliably for me, erroring out with
  RequestError: Request failed: success.
  
  I've pasted error passages from the ha agent and vdsm logs below.
  
  Any pointers?
 
 looks like a VDSM bug, Dan?

Why? The exception is raised from deep inside the ovirt_hosted_engine_ha
code.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users