Re: [Users] Unable to attach to storage domain (Ovirt 3.2)

Sun, 22 Sep 2013 06:49:02 -0700

We actually got it working. Both of us were tired from working late, so we didn't find that the missing storage domain was actually one of the NFS exports for awhile. After removing our NFS ISO domain and NFS export domain everything came up. 

Dan

On 9/22/13 6:08 AM, Ayal Baron wrote:

If I understand correctly you have a storage domain which is built of multiple 
(at least 2) LUNs.
One of these LUNs seems to be missing (Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z 
is an LVM PV UUID).
It looks like you are either not fully connected to the storage server (missing 
a connection) or the LUN mapping in LIO has been changed or that the chap 
password has changed or something.

LVM is able to report the LVs since the PV which contains the metadata is still 
accessible (which is also why you see the VG and why LVM knows that the 
Wy3Ymi... device is missing).

Can you compress and attach *all* of the vdsm.log* files?

----- Original Message -----

Hi Dan, it looks like one of the domains is missing:

6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50

Is there any target missing? (disconnected or somehow faulty or
unreachable)

--
Federico

----- Original Message -----

From: "Dan Ferris" <[email protected]>
To: [email protected]
Sent: Friday, September 20, 2013 4:01:06 AM
Subject: [Users] Unable to attach to storage domain (Ovirt 3.2)

Hi,

This is my first post to the list.  I am happy to say that we have been
using Ovirt for 6 months with a few bumps, but it's mostly been ok.

Until tonight that is...

I had to do a maintenance that required rebooting both of our Hypervisor
nodes.  Both of them run Fedora Core 18 and have been happy for months.
   After rebooting them tonight, they will not attach to the storage.  If
it matters, the storage is a server running LIO with a Fibre Channel
target.

Vdsm log:

Thread-22::DEBUG::2013-09-19
21:57:09,392::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd
iflag=direct if=/dev/b358e46b-635b-4c0e-8e73-0a494602e21d/metadata
bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2013-09-19
21:57:09,400::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> =
'1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied,
0.000547161 s, 7.5 MB/s\n'; <rc> = 0
Thread-23::DEBUG::2013-09-19
21:57:16,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' got the operation mutex
Thread-23::DEBUG::2013-09-19
21:57:16,587::misc::84::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo
-n /sbin/lvm vgs --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 filter = [
\\"a%360014055193f840cb3743f9befef7aa3%\\", \\"r%.*%\\" ] }  global {
locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {
retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix
--separator | -o
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free
6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50' (cwd None)
Thread-23::DEBUG::2013-09-19
21:57:16,643::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> =
'  Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found\n';
<rc> = 5
Thread-23::WARNING::2013-09-19
21:57:16,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
['  Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found']
Thread-23::DEBUG::2013-09-19
21:57:16,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' released the operation mutex
Thread-23::ERROR::2013-09-19
21:57:16,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
monitoring information
Traceback (most recent call last):
    File "/usr/share/vdsm/storage/domainMonitor.py", line 182, in
_monitorDomain
      self.domain = sdCache.produce(self.sdUUID)
    File "/usr/share/vdsm/storage/sdc.py", line 97, in produce
      domain.getRealDomain()
    File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
      return self._cache._realProduce(self._sdUUID)
    File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
      domain = self._findDomain(sdUUID)
    File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
      raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50',)

vgs output (Note that I don't know what the device
(Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is) :

[root@node01 vdsm]# vgs
    Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
    VG                                   #PV #LV #SN Attr   VSize   VFree
    b358e46b-635b-4c0e-8e73-0a494602e21d   1  39   0 wz--n-   8.19t  5.88t
    build                                  2   2   0 wz-pn- 299.75g 16.00m
    fedora                                 1   3   0 wz--n- 557.88g     0

lvs output:

[root@node01 vdsm]# lvs
    Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
    LV                                   VG
       Attr      LSize    Pool Origin Data%  Move Log Copy%  Convert
    0b8cca47-313f-48da-84f2-154810790d5a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    0f6f7572-8797-4d84-831b-87dbc4e1aa48
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    19a1473f-c375-411f-9a02-c6054b9a28d2
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   50.00g
    221144dc-51dc-46ae-9399-c0b8e030f38a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    2386932f-5f68-46e1-99a4-e96c944ac21b
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    3e027010-931b-43d6-9c9f-eeeabbdcd47a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    2.00g
    4257ccc2-94d5-4d71-b21a-c188acbf7ca1
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  200.00g
    4979b2a4-04aa-46a1-be0d-f10be0a1f587
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    4e1b8a1a-1704-422b-9d79-60f15e165cb7
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    70bce792-410f-479f-8e04-a2a4093d3dfb
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    791f6bda-c7eb-4d90-84c1-d7e33e73de62
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    818ad6bc-8da2-4099-b38a-8c5b52f69e32
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  120.00g
    861c9c44-fdeb-43cd-8e5c-32c00ce3cd3d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    86b69521-14db-43d1-801f-9d21f0a0e00f
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    8a578e50-683d-47c3-af41-c7e508d493e8
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    90463a7a-ecd4-4838-bc91-adccf99d9997
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    9170a33d-3bdf-4c15-8e6b-451622c8093b
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   80.00g
    964b9e32-c1ee-4152-a05b-0c43815f5ea6
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    975c0a26-f699-4351-bd27-dd7621eac6bd
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    9ec24f39-8b32-4247-bfb4-4b7f2cf86d9d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    a4f303bf-6e89-43c3-a801-046920cb24d6
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    a7115874-0f1c-4f43-ab3a-a6026ad99013
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    b1fb5597-a3bb-4e4b-b73f-d1752cc576cb
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    bc28d7c6-a14b-4398-8166-ac2f25b17312
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    bc72da88-f5fd-4f53-9c2c-af2fcd14d117
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    c2c1ba71-c938-4d71-876a-1bba89a5d8a9
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    c54eb342-b79b-45fe-8117-aab7137f5f9d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    c892f8b5-fadc-4774-a355-32655512a462
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    c9f636ce-efed-495d-9a29-cfaac1f289d3
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    cb657c62-44c8-43dd-8ea2-cbf5927cff72
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    cdba05ac-5f68-4213-827b-3d3518c67251
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    d98f3bfc-55b0-44f7-8a39-cb0920762cba
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
    e0708bc4-19df-4d48-a0e7-682070634dea
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
    ids
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-ao---  128.00m
    inbox
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  128.00m
    leases
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    2.00g
    master
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    1.00g
    metadata
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  512.00m
    outbox
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  128.00m
    root                                 build
       -wi-----p  298.74g
    swap_1                               build
       -wi-----p 1020.00m
    home                                 fedora
       -wi-ao---  503.88g
    root                                 fedora
       -wi-ao---   50.00g
    swap                                 fedora
       -wi-ao---    4.00g


The strange thing is that I can see all of the LVM volumes for the VMs.
   Both servers see the storage just fine.  This error has me baffled and
it's a total show stopper since the cluster will not come back up.

If anyone can help, that would be very appreciated.

Thanks,

Dan
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users


_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users


_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users

Reply via email to