Hi. Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message "ibv_create_qp err 13"? I'm having issues with this on a IBM ESS system.
The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to. The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs. MPI-jobs on the client nodes can communicate over the infiniband fabric without issues. Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue. When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted interfaces. I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this. Any help or suggestions is appreciated. Regards, Andreas Mattsson _____________________________________________ [cid:[email protected]] Andreas Mattsson Systems Engineer MAX IV Laboratory Lund University P.O. Box 118, SE-221 00 Lund, Sweden Visiting address: Fotongatan 2, 225 94 Lund Mobile: +46 706 64 95 44 [email protected]<mailto:[email protected]> www.maxiv.se<http://www.maxiv.se/>
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
