Hello everyone,
I recently found about Cloudstack and decided to give it a try
and I really like it so far but I bumped into a very weird config issue.
Here is my test environment.
10.10.10.200 - XenServer 7.1 host (all free patches installed,
ntp synced) eth0 is the management and also the only interface connected to the
network. There are no bonds.
10.10.10.201 - Centos 7 NFS server:
- firewalld disabled
- /etc/exports is:
/cloud_storage/isos *(rw,async,no_root_squash,no_subtree_check)
/cloud_storage/vms *(rw,async,no_root_squash,no_subtree_check)
10.10.10.210 - Cloudstack management server (OS is centos 7,
ntp synced)
Management network range - 10.10.10.202-209
Guest network range - 10.10.10.211-219
Storage network range - no ips set.
Primary storage is set to use the XenServer
host local storage.
10.10.10.150 - my PC address in the test network
(10.10.10.0/24, gate 10.10.10.1)
Zone, pod and cluster and host are created successfully.
But when I try to add secondary NFS server in Cloustack the operation fails.
Going through logs I see the following highlights (full log snipets are
attached to this mail):
Cloudstack-server.log -
Line 193: 2018-08-21 11:04:08,865 WARN
[c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-413:ctx-37d82b72)
(logid:fac8aabf) createFileSR failed! due to the following:
SR_BACKEND_FAILURE_61File SR creation error [opterr=fail to mount FileSR. Errno
is 32]
Line 221: 2018-08-21 11:04:08,865 WARN
[c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-413:ctx-37d82b72)
(logid:fac8aabf) Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template due to
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
the following: SR_BACKEND_FAILURE_61File SR creation error [opterr=fail to
mount FileSR. Errno is 32]
XenServer SMlog -
Aug 21 11:04:08 cloudstack-office SM: [20645] ['mount', '--bind',
'/var/cloud_mount/12fb334c-64e6-3206-9373-7ae0627f83d2/template/tmpl/1/1',
'/var/run/sr-mount/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6']
Aug 21 11:04:08 cloudstack-office SM: [20645] FAILED in util.pread: (rc 32)
stdout: '', stderr: 'mount: mount
/var/cloud_mount/12fb334c-64e6-3206-9373-7ae0627f83d2/template/tmpl/1/1 on
/run/sr-mount/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6 failed: Stale file handle
Aug 21 11:04:08 cloudstack-office SM: [20645] '
Aug 21 11:04:08 cloudstack-office SM: [20645] Raising exception [61, File SR
creation error [opterr=fail to mount FileSR. Errno is 32]]
Aug 21 11:04:08 cloudstack-office SM: [20645] lock: released
/var/lock/sm/f517ce41-148e-4f88-9bdc-15d7e2d0eaa6/sr
Aug 21 11:04:08 cloudstack-office SM: [20645] ***** generic exception:
sr_attach: EXCEPTION <class 'SR.SROSError'>, File SR creation error
[opterr=fail to mount FileSR. Errno is 32]
XenSever -XenSource.log -
Aug 21 11:05:38 cloudstack-office xapi: [debug|cloudstack-office|109042 UNIX
/var/lib/xcp/xapi|dispatch:SR.add_to_other_config D:e87a119cfb36|api_effect]
SR.add_to_other_config
Aug 21 11:05:38 cloudstack-office xapi: [ info|cloudstack-office|109033
|sm_exec D:143bd363be92|xapi] Session.destroy
trackid=a5d69dd0a4351c2751078822b0a8844b
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033
|SR.attach D:070d838f0344|backtrace] sm_exec D:143bd363be92 failed with
exception Storage_interface.Backend_error(_)
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033
|SR.attach D:070d838f0344|backtrace] Raised Storage_interface.Backend_error(_)
Aug 21 11:05:38 cloudstack-office xapi: [error|cloudstack-office|109033
|SR.attach D:070d838f0344|backtrace] 1/8 xapi @ cloudstack-office Raised at
file sm_exec.ml, line 216
What is weird is that I am able to mount the NFS through XenServer -> new
storage -> NFS wizard (using both NFSv3 and NFSv4 protocols). When added I am
able to create disks (a.k.a. the host has write access).
Some NFS connection/accessibility tests were run from the XenServer hosts with
the following results:
- RPC test
[root@cloudstack-office tmp]# rpcinfo -t 10.10.10.201 nfs
program 100003 version 3 ready and waiting
program 100003 version 4 ready and waiting
- Mount scan
[root@cloudstack-office /]# showmount -e 10.10.10.201
Export list for 10.10.10.201:
/cloud_storage/vms *
/cloud_storage/isos *
Note: both of those work manually but not through cloudstack
- NFS sr-proble returns this for both shares:
[root@cloudstack-office tmp]# xe sr-probe type=nfs
device-config:server=10.10.10.201 device-config:serverpath=/cloud_storage/vms
<?xml version="1.0" ?>
<SRlist/>
Any idea why the NFS cannot be added as secondary storage :(?
I am so out of ideas. Sorry for the long mail.
Best regards,
Jordan Kostov
<<attachment: cloudstack_and_xen_logs.zip>>
