Hi Yves, The specific issue with using both IP and IB based communication, in my understanding, is the polling of the TCP sockets that introduces some delay in receiving other protocol data. This situation prevents IB (and other high-speed interconnects) from achieving the same performance as when only IB is used. In fact, you should see a configure warning to this effect when enabling both IB and TCP. However, the performance will not be brought down to the level of the slowest connection (1GbE in your case) it will be some percentage slower.
I don't follow what you mean with using fibre channel between servers as an alternative. The idea is that all servers (from the OrangeFS perspective) mus tbe able to communicate over IB and IP (over 1GbE). Clients with IB would use IB to talk to the servers, clients with only 1GbE would use IP to talk to the servers. If you try the dual IP and IB configuration, I think it'd be helpful information for other users if you could post the performance difference you saw between IB-only and dual IB and IP in your setup. Thanks, Michael On Wed, Oct 19, 2011 at 8:30 AM, Becky Ligon <[email protected]> wrote: > You will need to test in your environment. Depending on your workload, it > may be just fine. > > Becky > > > On Wed, Oct 19, 2011 at 8:10 AM, Yves Revaz <[email protected]> wrote: > >> ** >> On 10/19/2011 01:46 PM, Becky Ligon wrote: >> >> In this situation, the best solution is to setup OrangeFS using TCP/IP, >> which is the default communication protocol, and setup infiniband as IB over >> TCP. >> >> OrangeFS is capable of understanding IB and TCP at the same time; >> however, there is a performance hit. So, in a production environment, you >> don't want to run OrangeFS in this way. >> >> >> Thanks for your reply. >> It did different experiment using either IB or TCP. With TCP through 1Gbit >> ethernet (linking the pvfs2 servers together), >> the performance is a bit disappointing, while it is excellent with IB. >> Unfortunately our cluster >> has only one IB network and we wanted to use it for MPI communication >> only. >> So, if I do understand well, there is no way to overcome this problem, >> except using IB over TCP, which is not recommended... >> What will be the penality of using it ? Does it reduce the performace to >> the one obtained with TCP through 1Gbit ethernet ? >> >> What about using fiber channel betwen servers ? >> >> Cheers, >> >> yves >> >> >> >> >> Becky >> >> On Tue, Oct 18, 2011 at 5:35 PM, Yves Revaz <[email protected]> wrote: >> >>> Hi all, >>> >>> I would like to set up a parallel file system using orangefs, >>> composed of 4 nodes (acting as metadata and io servers) >>> using and infiniband network. >>> >>> Then, is it possible to reach this parallel file system from another >>> node, that does not share the infiniband network but has only >>> a Gbit network ? (obviously all 4 nodes share also this Gbit network). >>> >>> Thanks in advance, >>> >>> yves >>> >>> >>> -- >>> (o o) >>> --------------------------------------------oOO--(_)--OOo------- >>> Yves Revaz >>> Laboratory of Astrophysics EPFL >>> Observatoire de Sauverny Tel : ++ 41 22 379 24 28 >>> 51. Ch. des Maillettes Fax : ++ 41 22 379 22 05 >>> 1290 Sauverny e-mail : [email protected] >>> SWITZERLAND Web : http://www.lunix.ch/revaz/ >>> ---------------------------------------------------------------- >>> >>> _______________________________________________ >>> Pvfs2-users mailing list >>> [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>> >> >> >> >> -- >> Becky Ligon >> OrangeFS Support and Development >> Omnibond Systems >> Anderson, South Carolina >> >> >> >> >> -- >> (o o) >> --------------------------------------------oOO--(_)--OOo------- >> Dr. Yves Revaz >> Laboratory of Astrophysics >> Ecole Polytechnique Fédérale de Lausanne (EPFL) >> Observatoire de Sauverny Tel : ++ 41 22 379 24 28 >> >> 51. Ch. des Maillettes Fax : ++ 41 22 379 22 05 >> 1290 Sauverny e-mail : [email protected] >> SWITZERLAND Web : http://www.lunix.ch/revaz/ >> ---------------------------------------------------------------- >> >> > > > -- > Becky Ligon > OrangeFS Support and Development > Omnibond Systems > Anderson, South Carolina > > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
