Hi all, I am configuring an opteron cluster with connectX Infiniband. I have a problem that if I run one of the NAS tests, it works the first, and maybe 2nd time, but after that the jobs instantly fail with messages like this-
[Rank 44][cm.c: line 860]poll CQ failed -2 [Rank 51][cm.c: line 860]poll CQ failed -2 [Rank 119][cm.c: line 860]poll CQ failed -2 [Rank 85][cm.c: line 860]poll CQ failed -2 [Rank 0][cm.c: line 860]poll CQ failed -2 [Rank 9][cm.c: line 860]poll CQ failed -2 [Rank 26][cm.c: line 860]poll CQ failed -2[Rank 43][cm.c: line 860] poll CQ failed -2 [Rank 94][cm.c: line 860]poll CQ failed -2 [Rank 111][cm.c: line 860]poll CQ failed -2 I can easily reproduce this with only 2 systems using a 16 process LU job, class B. Here are the configs I've tried- Suse 11 with distro provided IB driver and libraries,etc, using mvapich as provided by ohio state Suse 11 with distro driver, using OFED 1.3.1 libraries and mvapich Suse 10.3 with OFED 1.3.1, OFED 1.2.5.4, and OFED 1.4rc3 They all have the same basic problem. I think one of them reported "Error polling CQ" instead of "poll CQ failed". If I replace the connectX cards with regular DDR cards the problem goes away. I'm getting quite stumped at this point and would appreciate any suggestions or patches. Thanks, Rick -- Richard Warner Lead Systems Integrator Microway, Inc (508)732-5517 _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
