Hello, I have still problem with performance of OrangeFS over IB. We have servers with Intel True Scale HCA and using lastest Intel drivers (7.3.) Fabric is managed by OpenSM from Intel package. Now I can say that InfiniBand works.
I have support of BMI_Tcp disabled. I tried to run OrangeFS over Ethernet and it works fine (of course I had recompiled it with Tcp support, without openib support, with configuration changed). When I instlled OrangeFs with IB support, starts servers and try to mount the filesystem throw kernel module i finded error messages in /var/log/orangefs-server.log [D 08/27/2014 15:14:10] PVFS2 Server version 2.8.8-orangefs starting. [E 08/27/2014 15:14:43] Error: encourage_recv_incoming: unknown message header type 0 len 8. [E 08/27/2014 15:14:43] [bt] /usr/local/orangefs/sbin/pvfs2-server(error+0xca) [0x46d5ea] [E 08/27/2014 15:14:43] [bt] /usr/local/orangefs/sbin/pvfs2-server() [0x46a063] [E 08/27/2014 15:14:43] [bt] /usr/local/orangefs/sbin/pvfs2-server() [0x46c0a5] [E 08/27/2014 15:14:43] [bt] /usr/local/orangefs/sbin/pvfs2-server(BMI_testcontext+0xfb) [0x4238eb] [E 08/27/2014 15:14:43] [bt] /usr/local/orangefs/sbin/pvfs2-server() [0x41fddf] [E 08/27/2014 15:14:43] [bt] /lib64/libpthread.so.0(+0x79d1) [0x7ffec072f9d1] [E 08/27/2014 15:14:43] [bt] /lib64/libc.so.6(clone+0x6d) [0x7ffec0268b5d] [E 08/27/2014 15:31:51] When i tried to copy something to server throw pvfs-cp utils i didnt got error messages in server log, but the speed is still poor (about 7MB/s) Thanks for replays Hanousek Vít P.S.: for testing im using old berkleyDB (I was warn by configure script), but I think it shount slow the servers, isnt it? ---------- Původní zpráva ---------- Od: vithanousek Komu: Becky Ligon Datum: 20. 8. 2014 23:26:22 Předmět: Re: [Pvfs2-users] OrangeFS 2.8.8 slow on IB Thanks for replay, partially yes, exactly I used for InfiniBand support: ./configure --prefix=/usr/local/orangefs --with-kernel=/usr/src/kernels/2.6.32-431.17.1.el6.x86_64/ --with-openib=/usr/local/ofed/ --without-bmi-tcp --enable-shared Now, Im little bit confused ... by documentation, ib is for Mellanox IB, and openib is for OFED stack... We have Intel (Qlogic) HCA, so we should use openib, shouldnt we? But, I found that there are performance problems with comunication over IB with OpenMPI, so it looks like that InfiniBand is not set up correctly, but i dont know why, because tests like ib_read_bw, ibping, rping works, and shows good performance. Tomorrow I will try to install Intel IB driver suite, where should be more debuging tools for Intel IB HCA. Thanks for reply Hanousek Vít ---------- Původní zpráva ---------- Od: Becky Ligon Komu: vithanousek Datum: 20. 8. 2014 22:21:06 Předmět: Re: [Pvfs2-users] OrangeFS 2.8.8 slow on IB Did you reconfigure your installation with the --without-tcp as well as with --with-ib? Becky Sent from my iPhone > On Aug 20, 2014, at 7:48 AM, vithanousek wrote: > > Hello, > > I am trying to set up OrangeFS on our cluster, but I stuck in problems with > performance. > > When I installed OrangeFS only with Ethernet support, it works perfectly, the > bottle neck of bandwith is speed of disk and network (I getting cca 150MB/s) > > But when I installed OrangeFS only with InfiniBand support, it works, but i > got speed max about 5MB/s. > > Our cluster has 4x QDR Qlogic InfiniBand with Qlogic 12200 switch. We are > using CentOS 6.5 and we have instaled OFED software stack 3.12. But I'm not > sure if the problem is in InfiniBand or in OrangeFS. > What parts of OFED stack is needed to run OrangeFS over InfiniBand? > > Thanks for replays > Hanousek Vít > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
