To make a long story short, I made rdma client connect files and mounted with them directly :
#/etc/glusterd/vols/pirdist/pirdist.rdma-fuse.vol /pirdist glusterfs transport=rdma 0 0 #/etc/glusterd/vols/pirstripe/pirstripe.rdma-fuse.vol /pirstripe glusterfs transport=rdma 0 0 the transport=rdma does nothing here since it reads the parameters from .vol files . However you'll see that they're now commented out since RDMA has been very unstable for us. Servers lose their connections to each other, which somehow causes gbe clients to lose their connections. IP over IB however is working great, although at the expense of some performance vs RDMA, but it's still much better than gbe. On Thu, Jun 7, 2012 at 4:25 AM, bxma...@gmail.com <bxma...@gmail.com> wrote: > Hello, > > at first it was tcp then tcp,rdma. > > You are right that without tcp definition ".rdma" is not working. But > now i have another problem. > I'm trying tcp / rdma, im trying even tcp/rdma using normal network > card ( not using infiniband IP but normal 1gbit network > card and i have still same speed, upload about 30mb/s and download > about 200mb/s .. so i'm not sure if rdma is even working. > > Native infiniband is giving me 3500mb/s speed with benchmark tests > (ib_rdma_bw ). > > thanks > > Matus > > 2012/6/7 Amar Tumballi <ama...@redhat.com>: >> On 06/07/2012 02:04 PM, bxma...@gmail.com wrote: >>> >>> Hello, >>> >>> i have a problem with gluster 3.2.6 and infiniband. With gluster 3.3 >>> its working ok but with 3.2.6 i have following problems: >>> >>> when i'm trying to mount rdma volume using command mount -t glusterfs >>> 192.168.100.1:/atlas1.rdma mount i get: >>> >>> [2012-06-07 04:30:18.894337] I [glusterfsd.c:1493:main] >>> 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs >>> version 3.2.6 >>> [2012-06-07 04:30:18.907499] E >>> [glusterfsd-mgmt.c:628:mgmt_getspec_cbk] 0-glusterfs: failed to get >>> the 'volume file' from server >>> [2012-06-07 04:30:18.907592] E >>> [glusterfsd-mgmt.c:695:mgmt_getspec_cbk] 0-mgmt: failed to fetch >>> volume file (key:/atlas1.rdma) >>> [2012-06-07 04:30:18.907995] W [glusterfsd.c:727:cleanup_and_exit] >>> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xc9) >>> [0x7f784e2c8bc9] (-->/usr/local/ >>> lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f784e2c8975] >>> (-->/usr/local/sbin/glusterfs(mgmt_getspec_cbk+0x28b) [0x40861b]))) >>> 0-: received signum (0) >>> , shutting down >>> [2012-06-07 04:30:18.908049] I [fuse-bridge.c:3727:fini] 0-fuse: >>> Unmounting 'mount'. >>> >>> Same command without ".rdma" works ok. >>> >> >> Is the volume's transport type only 'rdma' ? or 'tcp,rdma' ? If its only >> 'rdma', then appending ".rdma" to volume name is not required. The appending >> of ".rdma" is only required when there are both type of transports on a >> volume (ie, 'tcp,rdma'), as from the client you can decide which transport >> you want to mount. >> >> default volume name would point to 'tcp' transport type, and appending >> ".rdma", will point to rdma transport type. >> >> Hope that is clear now. >> >> Regards, >> Amar > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
