Re: [ovirt-users] Emergency. ovirt-engine won't start after upgrade to 3.6.1
Thanks for the quick respond. May I know where I can find the "log-collector" report? selinux is set to permissive on the machine. That should rule out selinux issue, correct? I ran ovirt-engine startup under strace & generated the dump. I added that to the bug entry https://bugzilla.redhat.com/show_bug.cgi?id=1292544 Please see if the output has anything useful identifying the root cause. Thanks, Eric From: Yedidyah Bar David <d...@redhat.com> Sent: 25 January 2016 14:56 To: Sandro Bonazzola; Klein, Gil Cc: Eric Wong; users Subject: Re: [ovirt-users] Emergency. ovirt-engine won't start after upgrade to 3.6.1 On Mon, Jan 25, 2016 at 8:40 AM, Sandro Bonazzola <sbona...@redhat.com> wrote: > Can you please provide a full sos or log-collector report? We already got one such report [1], but didn't manage to understand how it happened, and the machine was later reinstalled. Sandro - it does have a sosreport attached, perhaps you want to take a look too. Eric - please also check audit (selinux) logs, and please also try running under strace, something like: Open shell1: # echo $$ note the pid, and then: Open shell2: # strace -f -o /tmp/engine-trace -s 4096 -tt -p PID (where PID is the pid of shell1) Then on shell1: # service ovirt-engine start Then please attach the trace to bug [1]. Thanks! [1] https://bugzilla.redhat.com/show_bug.cgi?id=1292544 > > Il 25/Gen/2016 06:58, "Eric Wong" <eric.w...@solvians.com> ha scritto: >> >> All oVirt guru: >> >> >> >> Please help. After upgrading our 3.5 install to the latest 3.6.1, we are >> not able to start ovirt-engine anymore. We completely lost control to our >> virtualization environment at this point. >> >> >> >> There is nothing being logged in server.log & engine.log. The only entry >> I saw in /var/log/messages is: >> >> >> >> Ovirt-engine: ERROR run:352 Error: process terminated with status code 1. >> >> >> >> Can anyone help us at least to start the ovirt-engine once again? The >> enclosed is the upgrade log L >> >> >> >> Thanks, >> >> Eric >> >> >> >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Archiving huge ovirt_engine_history table
Hello oVirt guru out there: I notice our oVirt engine postgres db size is growing quite fast for past couple of months. I checked the database size. Found that our ovirt_engine_history is 73GB in size. engine=# \connect ovirt_engine_history You are now connected to database "ovirt_engine_history" as user "postgres". ovirt_engine_history=# SELECT pg_size_pretty( pg_database_size( current_database() ) ) As human_size , pg_database_size( current_database() ) As raw_size; human_size | raw_size +- 73 GB | 78444213368 (1 row) Brief check the records, there are entries dated back 2014. I want to see if there is a safe way to archive and remove some of the older records? Thanks, Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] huge ovirt_engine_history
Hello oVirt guru out there: I notice our oVirt engine postgres db size is growing quite fast for past couple of months. I checked the database size. Found that our ovirt_engine_history is 73GB in size. engine=# \connect ovirt_engine_history You are now connected to database "ovirt_engine_history" as user "postgres". ovirt_engine_history=# SELECT pg_size_pretty( pg_database_size( current_database() ) ) As human_size , pg_database_size( current_database() ) As raw_size; human_size | raw_size +- 73 GB | 78444213368 (1 row) Brief check the records, there are entries dated back 2014. I want to see if there is a safe way to archive and remove some of the older records? Thanks, Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Clean up Hosted Engine entry in oVirt console
Didi: Thanks for taking the time. Let me try to provide more details. I added my comments below your questions. Many Thanks, Eric -Original Message- From: Yedidyah Bar David [mailto:d...@redhat.com] Sent: Wednesday, April 15, 2015 4:00 PM To: Eric Wong Cc: users@ovirt.org Subject: Re: [ovirt-users] Clean up Hosted Engine entry in oVirt console - Original Message - From: Eric Wong eric.w...@solvians.com To: users@ovirt.org Sent: Wednesday, April 15, 2015 10:41:08 AM Subject: [ovirt-users] Clean up Hosted Engine entry in oVirt console Hello oVirt Forum members: We are currently using oVirt Version 3.5.0.1-1.el6. We tried to migrate to use hosted-engine model. Somehow the hosted-engine got created, but the migration failed. Please give more details, it's not completely clear what you did. Let's define: oldenginemachine - the machine where you ran the engine before migration attempt Our oVirt environment is configured as followed: - 2 racks. Each rack has 10 blades with SAN storage. - the 2 racks cannot access the SAN storage of the other rack. So the X side blades can only reach X side SAN. D side blades can only see D side SAN. - Application VMs are running on both side of the rack. So in case one side of the rack completely failed, we still have the other side providing the application services. - Our oVirt management environment is also configured the same way. X side D side has their only ovirt-engine. ovirt-engine is running on CentOS VMs. Ovirt-engine on each side can see the blades and VMs on both sides. hostedenginehost - the host where you tried to run hosted-engine --deploy We tried to deploy hosted-engine on one of the blades on D side. We used this doc as our guide. http://www.ovirt.org/Migrate_to_Hosted_Engine newenginemachine - the vm created by 'hosted-engine --deploy' where you were supposed to setup an engine (by migrating data from oldenginemachine) On that D side blade, we got to the end of Part One of that document in which the newenginemachine seem to be created but I can never connect to it with vnc or remote-viewer (no firewall blocking). The hosted-engine is configured to use iSCSI storage instead of NFS. Was hostedenginehost registered in oldenginemachine before? Or is it a new host? I think the installation process automatically registered the hostedenginehost in the oldenginemachine during the Part One of the setup process. It was not there before. Was the engine there down while you tried migrating? The engine you talked about here. Is that the new or old one? The new hostedenginehost cannot be started. The oldenginemachine still runs. When I tried to started the HostedEngine (hostedenginehost) from the ovirt console, it tells me cannot be done because it is not managed by this ovirt host. Is there a chance that it was up, and 'hosted-engine --deploy' connected to it and tried doing stuff there? So I ran engine-setup to remove the installation. Where? I ran that on the blade where I initially ran the hosted-engine --deploy However, I am now leave with a VM entry in our oVirt console called HostedEngine. You mean, in oldenginemachine? That is correct. From the GUI provided by the oldenginemachine, I can see the entry HostedEngine. Tried to remove the entry by clicking on the Remove button on the console gives a message: Error while executing action: HostedEngine: Cannot remove VM. This VM is not managed by the engine. Indeed. As far as I can tell, all files that were touched by the hosted-engine are reverted back to before install state. So I am not sure where that HostedEngine entry is coming from. Is that a record in the database that I need to clean up manually from the db? Indeed. Not sure exactly how, but first let's be sure what the problem is. Best, -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Clean up Hosted Engine entry in oVirt console
Hello oVirt Forum members: We are currently using oVirt Version 3.5.0.1-1.el6. We tried to migrate to use hosted-engine model. Somehow the hosted-engine got created, but the migration failed. So I ran engine-setup to remove the installation. However, I am now leave with a VM entry in our oVirt console called HostedEngine. Tried to remove the entry by clicking on the Remove button on the console gives a message: Error while executing action: HostedEngine: Cannot remove VM. This VM is not managed by the engine. As far as I can tell, all files that were touched by the hosted-engine are reverted back to before install state. So I am not sure where that HostedEngine entry is coming from. Is that a record in the database that I need to clean up manually from the db? Any advice is appreciated. Thanks, Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Hosted Engine config in our hardware environment
Simone: I discussed with our SAN guy. We currently is not able to configure the SAN in D1 X1 to be seen by blades in both sides. So we are pretty much stuck with our current ovirt mgmt. vms configuration. I guess the main concern is how to control all the virtual machines with oVirt infrastructure if both our ovirt-mgmt vms went down, how can we start them up again. That also applies to any virtual machines. If the oVirt mgmt infrastructure is not able, I think there is no way to start a VM, correct? Again, Many Thanks, Eric -Original Message- From: Simone Tiraboschi [mailto:stira...@redhat.com] Sent: Tuesday, March 24, 2015 5:11 PM To: Eric Wong Cc: users@ovirt.org Subject: Re: [ovirt-users] Hosted Engine config in our hardware environment - Original Message - From: Eric Wong eric.w...@solvians.com To: Simone Tiraboschi stira...@redhat.com Cc: users@ovirt.org Sent: Tuesday, March 24, 2015 9:36:11 AM Subject: RE: [ovirt-users] Hosted Engine config in our hardware environment Simone: Thanks for your quick respond. It clarified some of the confusion I have with the hosted-engine. However, I still have one question. Sorry. It is my bad that I did not explain clearer on our hardware configuration, mainly on the SAN. We also have 2 SAN, one on D1 another one on X1. The blades on D1 can only see the storage in SAN D1. X1 blades can only see SAN X1. Does it mean that we will need to manage our oVirt environment separately when we switch to use hosted-engine? Because from your explanation, the hosted-engine in selected nodes will use separate iSCSI hosted-engine storage to store the DB. With hosted engine not only the DB but all the engine VM is going to be stored on a dedicated iSCSI LUN so each host involved (not managed by) in hosted-engine should be able to access that device in order to be able to start the engine VM after other host failures. So, if you want to have it restarting the engine VM on a X1 host after a complete failure of D1 rack, you need to store the engine VM on a location that is accessible both by D1 and X1. NFS could be an option too. Otherwise you should check your storage path design in order to have a cross rack path; you could also evaluate iSCSI multipathing. Since there is no cross D1 X1 storage in our current config, that means we need separate hosted-engine setup on each side. Thanks, Eric -Original Message- From: Simone Tiraboschi [mailto:stira...@redhat.com] Sent: Friday, March 20, 2015 7:06 PM To: Eric Wong Cc: users@ovirt.org Subject: Re: [ovirt-users] Hosted Engine config in our hardware environment - Original Message - From: Eric Wong eric.w...@solvians.com To: users@ovirt.org Sent: Thursday, March 19, 2015 7:04:04 PM Subject: [ovirt-users] Hosted Engine config in our hardware environment Hello oVirt guru out there: I want to seek some advice on upgrade path for our oVirt management vm configuration. We have been using oVirt for over 3 years. When we first setup oVirt environment, Hosted Engine componment did not exist. Our question is should we migrate our current configuration to use Hosted Engine? First let me give an overview of our configuration. We have blade servers in 2 separate racks. D1 X1. Each side has 10 blades. Storage is iSCSI SAN. Inside our oVirt 3.5.0.1-1.el6 installation, it is configured with 2 data centers. D1 X1. Each datacenter has the 10 blades for that side. The management function of oVirt (oVirt web console) is running off 2 VMs, ovirt-mgmt-1 on D1, and ovirt-mgmt-2 on X1. We have keepalived to maintain a flowing IP for the oVirt management console. The keepalived script makes sure only one copy of ovirt-engine is running at any time. It can be on D1 or X1. The mgmt VMs have Postgresql setup in replication mode. In case one of the mgmt vm failed, the other mgmt vm on the other rack can pick up the mgmt role. Both mgmt VMs can see all blades and SAN resources on D1 X1. This configuration has been working well for us. The drawback is if both ovirt mgmt vm crashed, we will not be able to start them or make any change to the ovirt environment. It is because the mgmt VMs are running within the oVirt domain. We tried to upgrade our configuration to Hosted Engine configuration. From what I understand, the Hosted Engine will run in a separate storage domain. In both times we tried to upgrade to Hosted Engine, they both failed during export and import of current configuration. Here you can find some hint about how to migrate to hosted-engine: http://www.ovirt.org/Migrate_to_Hosted_Engine I think my questions are: - will the Hosted Engine model works in our hardware configuration. With hardware in 2 racks, D1 X1. Can a single Hosted Engine manage hardware on both sides? - How can we achieve redundancy when running Hosted Engine? We need to have
Re: [ovirt-users] Hosted Engine config in our hardware environment
Simone: Thanks for your quick respond. It clarified some of the confusion I have with the hosted-engine. However, I still have one question. Sorry. It is my bad that I did not explain clearer on our hardware configuration, mainly on the SAN. We also have 2 SAN, one on D1 another one on X1. The blades on D1 can only see the storage in SAN D1. X1 blades can only see SAN X1. Does it mean that we will need to manage our oVirt environment separately when we switch to use hosted-engine? Because from your explanation, the hosted-engine in selected nodes will use separate iSCSI hosted-engine storage to store the DB. Since there is no cross D1 X1 storage in our current config, that means we need separate hosted-engine setup on each side. Thanks, Eric -Original Message- From: Simone Tiraboschi [mailto:stira...@redhat.com] Sent: Friday, March 20, 2015 7:06 PM To: Eric Wong Cc: users@ovirt.org Subject: Re: [ovirt-users] Hosted Engine config in our hardware environment - Original Message - From: Eric Wong eric.w...@solvians.com To: users@ovirt.org Sent: Thursday, March 19, 2015 7:04:04 PM Subject: [ovirt-users] Hosted Engine config in our hardware environment Hello oVirt guru out there: I want to seek some advice on upgrade path for our oVirt management vm configuration. We have been using oVirt for over 3 years. When we first setup oVirt environment, Hosted Engine componment did not exist. Our question is should we migrate our current configuration to use Hosted Engine? First let me give an overview of our configuration. We have blade servers in 2 separate racks. D1 X1. Each side has 10 blades. Storage is iSCSI SAN. Inside our oVirt 3.5.0.1-1.el6 installation, it is configured with 2 data centers. D1 X1. Each datacenter has the 10 blades for that side. The management function of oVirt (oVirt web console) is running off 2 VMs, ovirt-mgmt-1 on D1, and ovirt-mgmt-2 on X1. We have keepalived to maintain a flowing IP for the oVirt management console. The keepalived script makes sure only one copy of ovirt-engine is running at any time. It can be on D1 or X1. The mgmt VMs have Postgresql setup in replication mode. In case one of the mgmt vm failed, the other mgmt vm on the other rack can pick up the mgmt role. Both mgmt VMs can see all blades and SAN resources on D1 X1. This configuration has been working well for us. The drawback is if both ovirt mgmt vm crashed, we will not be able to start them or make any change to the ovirt environment. It is because the mgmt VMs are running within the oVirt domain. We tried to upgrade our configuration to Hosted Engine configuration. From what I understand, the Hosted Engine will run in a separate storage domain. In both times we tried to upgrade to Hosted Engine, they both failed during export and import of current configuration. Here you can find some hint about how to migrate to hosted-engine: http://www.ovirt.org/Migrate_to_Hosted_Engine I think my questions are: - will the Hosted Engine model works in our hardware configuration. With hardware in 2 racks, D1 X1. Can a single Hosted Engine manage hardware on both sides? - How can we achieve redundancy when running Hosted Engine? We need to have management capibility in case one side completely failed. Thanks in advance. Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Hosted Engine config in our hardware environment
Hello oVirt guru out there: I want to seek some advice on upgrade path for our oVirt management vm configuration. We have been using oVirt for over 3 years. When we first setup oVirt environment, Hosted Engine componment did not exist. Our question is should we migrate our current configuration to use Hosted Engine? First let me give an overview of our configuration. We have blade servers in 2 separate racks. D1 X1. Each side has 10 blades. Storage is iSCSI SAN. Inside our oVirt 3.5.0.1-1.el6 installation, it is configured with 2 data centers. D1 X1. Each datacenter has the 10 blades for that side. The management function of oVirt (oVirt web console) is running off 2 VMs, ovirt-mgmt-1 on D1, and ovirt-mgmt-2 on X1. We have keepalived to maintain a flowing IP for the oVirt management console. The keepalived script makes sure only one copy of ovirt-engine is running at any time. It can be on D1 or X1. The mgmt VMs have Postgresql setup in replication mode. In case one of the mgmt vm failed, the other mgmt vm on the other rack can pick up the mgmt role. Both mgmt VMs can see all blades and SAN resources on D1 X1. This configuration has been working well for us. The drawback is if both ovirt mgmt vm crashed, we will not be able to start them or make any change to the ovirt environment. It is because the mgmt VMs are running within the oVirt domain. We tried to upgrade our configuration to Hosted Engine configuration. From what I understand, the Hosted Engine will run in a separate storage domain. In both times we tried to upgrade to Hosted Engine, they both failed during export and import of current configuration. I think my questions are: - will the Hosted Engine model works in our hardware configuration. With hardware in 2 racks, D1 X1. Can a single Hosted Engine manage hardware on both sides? - How can we achieve redundancy when running Hosted Engine? We need to have management capibility in case one side completely failed. Thanks in advance. Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] Cannot set Compatibility Version to 3.3
Hello oVirt gurus out there: I recently upgraded our oVirt 3.1 installation to 3.3.3. We have 2 clusters configuration. In the QA cluster, in which there is only one host, I can switch the cluster's compatibility version from 3.1 to 3.3. However, on the production cluster, in which there are 10 hosts, I am not able to switch the cluster compatibility version at all. I opened up the Edit Cluster window. Select 3.3 from the Compatibility Version dropdown list. Click OK to save the change. But the Edit Cluster window just remain open. I checked all the logs in the ovirt-engine. Do not see anything obvious. All our hosts are running CentOS 6.5 vdsm-4.13.3-4. Can you guys provide some pointers as to where to check for failure? Thanks in advance. Eric ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
