Hi Dan, it looks like one of the domains is missing: 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
Is there any target missing? (disconnected or somehow faulty or unreachable) -- Federico ----- Original Message ----- > From: "Dan Ferris" <[email protected]> > To: [email protected] > Sent: Friday, September 20, 2013 4:01:06 AM > Subject: [Users] Unable to attach to storage domain (Ovirt 3.2) > > Hi, > > This is my first post to the list. I am happy to say that we have been > using Ovirt for 6 months with a few bumps, but it's mostly been ok. > > Until tonight that is... > > I had to do a maintenance that required rebooting both of our Hypervisor > nodes. Both of them run Fedora Core 18 and have been happy for months. > After rebooting them tonight, they will not attach to the storage. If > it matters, the storage is a server running LIO with a Fibre Channel target. > > Vdsm log: > > Thread-22::DEBUG::2013-09-19 > 21:57:09,392::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd > iflag=direct if=/dev/b358e46b-635b-4c0e-8e73-0a494602e21d/metadata > bs=4096 count=1' (cwd None) > Thread-22::DEBUG::2013-09-19 > 21:57:09,400::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = > '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, > 0.000547161 s, 7.5 MB/s\n'; <rc> = 0 > Thread-23::DEBUG::2013-09-19 > 21:57:16,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm > reload operation' got the operation mutex > Thread-23::DEBUG::2013-09-19 > 21:57:16,587::misc::84::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo > -n /sbin/lvm vgs --config " devices { preferred_names = > [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 > disable_after_error_count=3 filter = [ > \\"a%360014055193f840cb3743f9befef7aa3%\\", \\"r%.*%\\" ] } global { > locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { > retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix > --separator | -o > uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free > 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50' (cwd None) > Thread-23::DEBUG::2013-09-19 > 21:57:16,643::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = > ' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found\n'; > <rc> = 5 > Thread-23::WARNING::2013-09-19 > 21:57:16,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] > [' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found'] > Thread-23::DEBUG::2013-09-19 > 21:57:16,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm > reload operation' released the operation mutex > Thread-23::ERROR::2013-09-19 > 21:57:16,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) > Error while collecting domain 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50 > monitoring information > Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 182, in > _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 97, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) > StorageDomainDoesNotExist: Storage domain does not exist: > (u'6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50',) > > vgs output (Note that I don't know what the device > (Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is) : > > [root@node01 vdsm]# vgs > Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z. > VG #PV #LV #SN Attr VSize VFree > b358e46b-635b-4c0e-8e73-0a494602e21d 1 39 0 wz--n- 8.19t 5.88t > build 2 2 0 wz-pn- 299.75g 16.00m > fedora 1 3 0 wz--n- 557.88g 0 > > lvs output: > > [root@node01 vdsm]# lvs > Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z. > LV VG > Attr LSize Pool Origin Data% Move Log Copy% Convert > 0b8cca47-313f-48da-84f2-154810790d5a > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 0f6f7572-8797-4d84-831b-87dbc4e1aa48 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 19a1473f-c375-411f-9a02-c6054b9a28d2 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 50.00g > 221144dc-51dc-46ae-9399-c0b8e030f38a > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 2386932f-5f68-46e1-99a4-e96c944ac21b > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 3e027010-931b-43d6-9c9f-eeeabbdcd47a > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g > 4257ccc2-94d5-4d71-b21a-c188acbf7ca1 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 200.00g > 4979b2a4-04aa-46a1-be0d-f10be0a1f587 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 4e1b8a1a-1704-422b-9d79-60f15e165cb7 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 70bce792-410f-479f-8e04-a2a4093d3dfb > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 791f6bda-c7eb-4d90-84c1-d7e33e73de62 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 818ad6bc-8da2-4099-b38a-8c5b52f69e32 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 120.00g > 861c9c44-fdeb-43cd-8e5c-32c00ce3cd3d > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 86b69521-14db-43d1-801f-9d21f0a0e00f > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 8a578e50-683d-47c3-af41-c7e508d493e8 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 90463a7a-ecd4-4838-bc91-adccf99d9997 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > 9170a33d-3bdf-4c15-8e6b-451622c8093b > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 80.00g > 964b9e32-c1ee-4152-a05b-0c43815f5ea6 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 975c0a26-f699-4351-bd27-dd7621eac6bd > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > 9ec24f39-8b32-4247-bfb4-4b7f2cf86d9d > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > a4f303bf-6e89-43c3-a801-046920cb24d6 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > a7115874-0f1c-4f43-ab3a-a6026ad99013 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > b1fb5597-a3bb-4e4b-b73f-d1752cc576cb > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > bc28d7c6-a14b-4398-8166-ac2f25b17312 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > bc72da88-f5fd-4f53-9c2c-af2fcd14d117 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > c2c1ba71-c938-4d71-876a-1bba89a5d8a9 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > c54eb342-b79b-45fe-8117-aab7137f5f9d > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > c892f8b5-fadc-4774-a355-32655512a462 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > c9f636ce-efed-495d-9a29-cfaac1f289d3 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > cb657c62-44c8-43dd-8ea2-cbf5927cff72 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > cdba05ac-5f68-4213-827b-3d3518c67251 > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > d98f3bfc-55b0-44f7-8a39-cb0920762cba > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g > e0708bc4-19df-4d48-a0e7-682070634dea > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g > ids > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-ao--- 128.00m > inbox > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m > leases > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g > master > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 1.00g > metadata > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 512.00m > outbox > b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m > root build > -wi-----p 298.74g > swap_1 build > -wi-----p 1020.00m > home fedora > -wi-ao--- 503.88g > root fedora > -wi-ao--- 50.00g > swap fedora > -wi-ao--- 4.00g > > > The strange thing is that I can see all of the LVM volumes for the VMs. > Both servers see the storage just fine. This error has me baffled and > it's a total show stopper since the cluster will not come back up. > > If anyone can help, that would be very appreciated. > > Thanks, > > Dan > _______________________________________________ > Users mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
