Thanks Haim. Sorry it took so long to get back to the list. Firstly, I ended up sleeping on the problem, then realizing that it boils down to this:
. All VMs are down at this point . All storage domains in the DC were listed as down . I've got 3 hypervisors (2 of which I can put into maintenance mode) . The lone hypervisor has the right idea of what the master storage domain is supposed to be, and the database is wrong. So I stopped ovirt-engine, backed up the database to a sql file, and found the section that started with: -- TOC entry 3717 (class 0 OID 12577180) -- Dependencies: 193 -- Data for Name: storage_domain_static; Type: TABLE DATA; Schema: public; Owner: engine -- COPY storage_domain_static (id, storage, storage_name, storage_domain_type, storage_type, storage_domain_format_type, _create_date, _update_date, recoverable, last_time_used_as_master) FROM stdin; At that point I saw that BV4Pool1 had storage_domain value 0, and BVData-Master had a value of 1. I flipped them, restored the database (dropping the existing db), and rebooted the lone non-maintenance hypervisor. Then I started ovirt-engine and everything started to look rosy. If you do still, however, want logs, I'd be happy to provide them! -I On Sun, Dec 30, 2012 at 8:30 AM, Haim Ateya <[email protected]> wrote: > Hi Ian, > > I really like to understand how it happened, there are several ways to fix > this issue, > please send us engine and vdsm logs for a start. > as for fixing the issue, we will probably have to perform some manual > intervention on either engine data-base (preferable) or storage > metadata. > its a bit complex, and might cause other things to fail if not doing > carefully, so its your call. > > > ----- Original Message ----- > > From: "Ian Forde" <[email protected]> > > To: [email protected] > > Sent: Sunday, December 30, 2012 12:31:31 PM > > Subject: [Users] Fwd: Power outage = storage outage = ovirt crash with > mismatched master storage domain? > > > > > > (resend, as it might have tried to go through before my subscription > > to the list was active...) > > > > > > ---------- Forwarded message ---------- > > From: Ian Forde < [email protected] > > > Date: Sun, Dec 30, 2012 at 1:57 AM > > Subject: Power outage = storage outage = ovirt crash with mismatched > > master storage domain? > > To: [email protected] > > > > > > Hi all - > > > > > > I'm running Ovirt from the dreyou packages (ovirt-engine 3.1.0-3.26) > > on CentOS 6.3, with 3 hypervisors all running CentOS 6.3 with > > dreyou-packaged vdsm 4.10.1-0.77.20 running. Storage is a Synology > > 1812+. > > > > > > And tonight I had a series of power "events" that affected the NAS. > > (So much for the small UPS. Storage has now been moved to a bigger > > UPS.) Anyway - it turns out that my storage went down 3 times while > > I was out. The NAS recovered. Ovirt didn't. > > > > > > I've tried putting boxes into maintenance mode. I've tried reboots > > and "confirm reboot". I've even shut everything down and brought it > > back up from scratch. What I'm facing now is a message in the GUI > > that states: > > > > > > "Sync Error on Master Domain between Host bv-hv01 and oVirt Engine. > > Domain BV4Pool1 is marked as Master in oVirt Engine database but not > > on the Storage side. Please consult with Support on how to fix this > > issue." > > > > > > Now, what was *supposed* to be my master storage domain is named > > "BV4Data-Master", and BV4Pool1 is where the VM disks are stored. > > (I'm not crazy about putting VM disks into the master pool.) > > > > > > So why does the database have the wrong entry and how do I fix it? > > Currently everything is *down*, given that the last time I tried to > > mess with oVirt innards I had to recreate my DC and restore all of > > the VMs by hand. Not something I'd like to do again... > > > > > > (This, of course, also means that I need to get proper oVirt database > > backups going...) > > > > > > Any ideas? > > > > > > -I > > > > _______________________________________________ > > Users mailing list > > [email protected] > > http://lists.ovirt.org/mailman/listinfo/users > > >
_______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
