Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
- Original Message - From: Limor Gavish lgav...@gmail.com To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the storage domains table in the Data Centers tab in the GUI. In engine.log (attached) I see many errors. Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') vdsm.log should add more details about this exception Thanks Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
- Original Message - From: Limor Gavish lgav...@gmail.com To: Eli Mesika emes...@redhat.com Cc: users@ovirt.org, Yuval M yuva...@gmail.com, Nezer Zaidenberg nzaidenb...@mac.com Sent: Wednesday, March 20, 2013 12:47:49 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Thank you very much for your reply. I attached the vdsm.log thanks, I am CC people from our storage team. Ayal, Eduardo, Allon this is failing in clusterlock.py::acquireHostId Any idea on the problem origin? Thanks On Wed, Mar 20, 2013 at 11:52 AM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Limor Gavish lgav...@gmail.com To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the storage domains table in the Data Centers tab in the GUI. In engine.log (attached) I see many errors. Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') vdsm.log should add more details about this exception Thanks Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Also adding Federico to the thread, could be a san lock issue: SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') Regards, Maor On 03/20/2013 01:50 PM, Eli Mesika wrote: - Original Message - From: Limor Gavish lgav...@gmail.com To: Eli Mesika emes...@redhat.com Cc: users@ovirt.org, Yuval M yuva...@gmail.com, Nezer Zaidenberg nzaidenb...@mac.com Sent: Wednesday, March 20, 2013 12:47:49 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Thank you very much for your reply. I attached the vdsm.log thanks, I am CC people from our storage team. Ayal, Eduardo, Allon this is failing in clusterlock.py::acquireHostId Any idea on the problem origin? Thanks On Wed, Mar 20, 2013 at 11:52 AM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Limor Gavish lgav...@gmail.com To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the storage domains table in the Data Centers tab in the GUI. In engine.log (attached) I see many errors. Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') vdsm.log should add more details about this exception Thanks Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
- Original Message - From: Limor Gavish lgav...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Yuval M yuva...@gmail.com, users@ovirt.org, Nezer Zaidenberg nzaidenb...@mac.com Sent: Wednesday, March 20, 2013 11:47:49 AM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Thank you very much for your reply. I attached the vdsm.log Hi Limor, can you please inspect the status of the NFS mount? # mkdir /mnt/tmp # mount -t nfs your_nfs_share /mnt/tmp # cd /mnt/tmp/1902354b-4c39-4707-ac6c-3637aaf1943b/dom_md And please report the output of: # ls -l # sanlock direct dump ids Can you also include more vdsm logs? More specifically the ones where the NFS domain has been created? (createStorageDomain with sdUUID='1902354b-4c39-4707-ac6c-3637aaf1943b') Thanks, -- Federico ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
- Original Message - From: Limor Gavish lgav...@gmail.com To: Federico Simoncelli fsimo...@redhat.com Cc: Yuval M yuva...@gmail.com, users@ovirt.org, Nezer Zaidenberg nzaidenb...@mac.com, Eli Mesika emes...@redhat.com, Maor Lipchuk mlipc...@redhat.com Sent: Wednesday, March 20, 2013 9:02:35 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Thank you very much for your response. Attached VDSM logs as you requested (The VDSM logs where the NFS domain was created were missing so we had to recreate the NFS domain, therefore the sdUUID has changed). Here is the rest of the commands you asked: [root@bufferoverflow wil]# mount -t nfs bufferoverflow:/home/BO_Ovirt_Storage /mnt/tmp [root@bufferoverflow wil]# cd /mnt/tmp/1083422e-a5db-41b6-b667-b9ef1ef244f0/dom_md/ [root@bufferoverflow dom_md]# ls -l total 2052 -rw-rw 1 vdsm kvm 1048576 Mar 20 21:46 ids -rw-rw 1 vdsm kvm 0 Mar 20 21:45 inbox -rw-rw 1 vdsm kvm 2097152 Mar 20 21:45 leases -rw-r--r-- 1 vdsm kvm 311 Mar 20 21:45 metadata -rw-rw 1 vdsm kvm 0 Mar 20 21:45 outbox [root@bufferoverflow dom_md]# sanlock direct dump ids Sorry I should have mentioned that if you use root_squash for your nfs share you have to switch to the vdsm user: (root)# su -s /bin/sh vdsm (vdsm)$ cd /mnt/tmp/sduuid/dom_md/ (vdsm)$ sanlock direct dump ids (and now you should be able to see the output) If the output is still empty then used hexdump -C to inspect it (and eventually post it here compressed). Another important thing that you should check is: # ps fax | grep sanlock If the output doesn't look like the following: 1966 ?SLs0:00 wdmd -G sanlock 2036 ?SLsl 0:00 sanlock daemon -U sanlock -G sanlock 2037 ?S 0:00 \_ sanlock daemon -U sanlock -G sanlock Then I suggest you to update sanlock to the latest build: http://koji.fedoraproject.org/koji/buildinfo?buildID=377815 (sanlock-2.6-7.fc18) And eventually if after rebooting the problem persists, please post also the sanlock log (/var/log/sanlock.log) Please note, the VDSM is running as a system service (it was installed from a package) while ovirt-engine was built from sources and thus is not running as root. Is this an issue? It shouldn't be. -- Federico ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Updating from sanlock-2.6-4.fc18 to sanlock-2.6-7.fc18 fixed it. Huge thanks! On Mar 20, 2013 10:24 PM, Federico Simoncelli fsimo...@redhat.com wrote: - Original Message - From: Limor Gavish lgav...@gmail.com To: Federico Simoncelli fsimo...@redhat.com Cc: Yuval M yuva...@gmail.com, users@ovirt.org, Nezer Zaidenberg nzaidenb...@mac.com, Eli Mesika emes...@redhat.com, Maor Lipchuk mlipc...@redhat.com Sent: Wednesday, March 20, 2013 9:02:35 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2) Thank you very much for your response. Attached VDSM logs as you requested (The VDSM logs where the NFS domain was created were missing so we had to recreate the NFS domain, therefore the sdUUID has changed). Here is the rest of the commands you asked: [root@bufferoverflow wil]# mount -t nfs bufferoverflow:/home/BO_Ovirt_Storage /mnt/tmp [root@bufferoverflow wil]# cd /mnt/tmp/1083422e-a5db-41b6-b667-b9ef1ef244f0/dom_md/ [root@bufferoverflow dom_md]# ls -l total 2052 -rw-rw 1 vdsm kvm 1048576 Mar 20 21:46 ids -rw-rw 1 vdsm kvm 0 Mar 20 21:45 inbox -rw-rw 1 vdsm kvm 2097152 Mar 20 21:45 leases -rw-r--r-- 1 vdsm kvm 311 Mar 20 21:45 metadata -rw-rw 1 vdsm kvm 0 Mar 20 21:45 outbox [root@bufferoverflow dom_md]# sanlock direct dump ids Sorry I should have mentioned that if you use root_squash for your nfs share you have to switch to the vdsm user: (root)# su -s /bin/sh vdsm (vdsm)$ cd /mnt/tmp/sduuid/dom_md/ (vdsm)$ sanlock direct dump ids (and now you should be able to see the output) If the output is still empty then used hexdump -C to inspect it (and eventually post it here compressed). Another important thing that you should check is: # ps fax | grep sanlock If the output doesn't look like the following: 1966 ?SLs0:00 wdmd -G sanlock 2036 ?SLsl 0:00 sanlock daemon -U sanlock -G sanlock 2037 ?S 0:00 \_ sanlock daemon -U sanlock -G sanlock Then I suggest you to update sanlock to the latest build: http://koji.fedoraproject.org/koji/buildinfo?buildID=377815 (sanlock-2.6-7.fc18) And eventually if after rebooting the problem persists, please post also the sanlock log (/var/log/sanlock.log) Please note, the VDSM is running as a system service (it was installed from a package) while ovirt-engine was built from sources and thus is not running as root. Is this an issue? It shouldn't be. -- Federico ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
