Hello All, I'm having trouble mounting gluster volumes over RDMA. Everything seems to work fine until I try to mount the gluster volume. At this point it seems to crash glusterd and give me an error "could not create QP [Permission Denied]. If I change change the transport to tcp, everything works as expected. I am pretty new to Infiniband and RDMA, so this may very well be something I'm doing wrong. That said, I was able to get NFSoRDMA (and qperf) working on these same hosts, so I'm confident my hardware is working. This is on CentOS 7.5 using gluster 3.12.15 from the CentOS repository. I can't find anything regarding this specific error and any help is much appreciated.
# rdma works... [root@gluster1 ~]# qperf --use_bits_per_sec gluster2-ib rc_bi_bw rc_bi_bw: bw = 35.3 Gb/sec # NFSoRDMA works.... [root@gluster1 data]# mount -t nfs gluster2-ib:/test /mnt/test -o rdma,port=20049 [root@gluster1 ~]# [root@gluster1 data]# df -h|grep root /dev/mapper/centos_gluster1-root 33G 7.8G 26G 24% / gluster2-ib:/test 33G 29G 4.1G 88% /mnt/test [root@gluster1 ~]# gluster volume create data replica 3 transport rdma gluster1-ib:/zp1/data vmh2-ib:/zp1/data vmh3-ib:/zp1/data force volume create: data: success: please start the volume to access data [root@gluster1 ~]# [root@gluster1 ~]# gluster volume start data volume start: data: success [root@gluster1 ~]# [root@gluster1 ~]# gluster volume info data Volume Name: data Type: Replicate Volume ID: 6f32999c-2d20-4fc2-838b-82972171df7e Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: rdma Bricks: Brick1: gluster1-ib:/zp1/data Brick2: gluster2-ib:/zp1/data Brick3: gluster3-ib:/zp1/data Options Reconfigured: nfs.disable: on performance.client-io-threads: off [root@gluster1 ~]# [root@gluster1 ~]# mount -t glusterfs -o transport=rdma gluster1:/data /mnt/data Mount failed. Please check the log file for more details. [root@gluster1 ~]# [root@gluster1 ~]# gluster volume info data Connection failed. Please check if gluster daemon is operational. [root@gluster1 ~]# [root@gluster1 ~]# tail -n 30 /var/log/messages|grep glusterd Nov 10 11:18:07 gluster1 glusterd[1881]: [2018-11-10 16:18:07.457818] C [MSGID: 103021] [rdma.c:3263:gf_rdma_create_qp] 0-rdma.management: rdma.management: could not create QP [Permission denied] Nov 10 11:18:07 gluster1 glusterd[1881]: pending frames: Nov 10 11:18:07 gluster1 glusterd[1881]: patchset: git:// git.gluster.org/glusterfs.git Nov 10 11:18:07 gluster1 glusterd[1881]: signal received: 11 Nov 10 11:18:07 gluster1 glusterd[1881]: time of crash: Nov 10 11:18:07 gluster1 glusterd[1881]: 2018-11-10 16:18:07 Nov 10 11:18:07 gluster1 glusterd[1881]: configuration details: Nov 10 11:18:07 gluster1 glusterd[1881]: argp 1 Nov 10 11:18:07 gluster1 glusterd[1881]: backtrace 1 Nov 10 11:18:07 gluster1 glusterd[1881]: dlfcn 1 Nov 10 11:18:07 gluster1 glusterd[1881]: libpthread 1 Nov 10 11:18:07 gluster1 glusterd[1881]: llistxattr 1 Nov 10 11:18:07 gluster1 glusterd[1881]: setfsid 1 Nov 10 11:18:07 gluster1 glusterd[1881]: spinlock 1 Nov 10 11:18:07 gluster1 glusterd[1881]: epoll.h 1 Nov 10 11:18:07 gluster1 glusterd[1881]: xattr.h 1 Nov 10 11:18:07 gluster1 glusterd[1881]: st_atim.tv_nsec 1 Nov 10 11:18:07 gluster1 glusterd[1881]: package-string: glusterfs 3.12.15 Nov 10 11:18:07 gluster1 glusterd[1881]: ---------
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users