[EMAIL PROTECTED] wrote on Mon, 28 Aug 2006 17:24 +0100:
> I'm going to raise a different, although slightly related issue; please
> advise me if it should be sent off-list or create another "thread".
>
> I am also using Topspin IB PCI-X cards and TS120 switch, with Scientific
> Linux EL4.2 (2.6.9-11.ELsmp) on Opteron x86_64 - HP DL145G2 nodes -
> (Vikrant is SLES9 a typo? is it Scientific Linux EL?).
>
> IPoIB is running fine (I get around 680MB/s from netperf at 70% CPU usage)
> but as sson as I try to run anything that uses IB SDP (I think it uses
> RDMA), such as tests in topspin/bin/ or topspin/sbin/hca_stress, the
> machines where the test is run crash with an IB module oops
Personally I've been afraid to run SDP. It has been fraught with
problems in the past.
> So I am running PVFS but on the Gb boards; I don't want to use PVFS-IB
> while the IB stack is "unreliable"...
Note that PVFS-IB does not use IPoIB. It does not use SDP. It uses
native IB calls, much like an MPI implementation or native
InfiniBand program would use it. Thus I wouldn't be too worried
about stability problems. The kernel module part is completely
optional, so the worst that would happen if you tried PVFS-IB is
that your userspace program might crash.
> I have been advised to replace the Topspin/Cisco with the openIB stack,
> but I had yet no window to halt the nodes...
That seems to be the trend. We use both Mellanox IBGD-1.8.2 and
OpenIB here locally, for testing, and they work fine. The
programming interface for OpenIB is much nicer to work with though.
-- Pete
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
