Michael, Thanks for your suggestion. I had the same intuition as you, but then I used "iptraf" and saw no eth0 traffic associated with I/O on the Gluster volume. (the tool doesn't show the ib0 interface, unfortunately). node01 and node02 are entered into /etc/hosts, they resolve to the ipoib addresses and are pingable.
I will try increasing the number of threads and applying the patch Bryan suggested. Thanks, Adrian ________________________________________ From: Michael Mayer [[email protected]] Sent: 21 April 2012 10:21 To: Bryan Whitehead Cc: Ionescu, A.; [email protected] Subject: Re: [Gluster-users] Performance issues with striped volume over Infiniband Adrian, I think you should really try to restart from scratch, i.e. - setup ipoib with corresponding entries in DNS or /etc/hosts (make sure the IB hosts are pingable) - gluster peer probe on the ipoib hostnames - recreate volume using ipoib hostnames for the bricks For transport you can still use tcp (I usually set tcp,rdma to be flexible later on) I started naively myself a couple of weeks ago and did exactly the same than you assuming that IB transport would be autodetected. In fact, the 100 MB/s you are seeing indicated very strongly that any communication is still over Gigabit Ethernet (Which has a theoretical peak performance of 125 MB/s but given switch quality and protocol overhead you will reach more like 100-110 MB/s in reality) Michael. On 04/20/2012 07:42 PM, Bryan Whitehead wrote: > Max out the number of IO threads and apply a patch to make gluster > more agressive about spawning threads as in this thread: > > http://gluster.org/pipermail/gluster-users/2012-February/009590.html > > (the above thread is actually pretty good for getting performance out > of gluster with infiniband (I use IP over IB myself - could never get > the native rDMA to be fast enough). > > On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.<[email protected]> wrote: >> I tried this, but the performance is still the same (about ~100MB/s). >> >> A strange thing I noticed is that the node on which I run the benchmarking >> tool reports ~350 MB/s traffic on the loopback interface. >> >> Does anyone have any other suggestions? >> >> Thanks, >> Adrian >> ________________________________________ >> From: [email protected] [[email protected]] >> on behalf of Sabuj Pattanayek [[email protected]] >> Sent: 19 April 2012 22:32 >> To: [email protected] >> Subject: Re: [Gluster-users] Performance issues with striped volume over >> Infiniband >> >> On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.<[email protected]> wrote: >>> Thanks for your answer, Sabuj. However, I am not sure I understand what you >>> mean by trying with ipoib. >>> Do you mean specifying "transport tcp" and using the ipoib ips/hostnames >>> for the bricks? >>> If you manage to find a way to overcome this performance limitation, please >>> share with us your solution. >> Yes. >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > [email protected] > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list [email protected] http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
