There used to be issues with the CX-3 cards and specific ports for if you wanted to use IB and Eth, but that went away in later firmwares, as did a whole load of bits with it being slow to detect media type, so see if you are running an up to date Mellanox firmware (assuming it's a VPI card).
On CX-4 there is no auto detect media, but default is IB unless you changed it. Simon ________________________________________ From: [email protected] [[email protected]] on behalf of [email protected] [[email protected]] Sent: 18 June 2017 16:30 To: gpfsug main discussion list Subject: ?spam? Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Are any cards VPI that can do both eth and ib? I remember reading in documentation that that there is a bus order to having mixed media with mellanox cards. There is a module setting during init where you can set eth ib or auto detect. If the card is on auto it might be coming up eth and making the driver flake out because it's in the wrong order. Responding from my phone so I can't really look it up myself right now about what the proper order is, but maybe this might be some help troubleshooting. On Jun 18, 2017 12:58 AM, "Frank Tower" <[email protected]<mailto:[email protected]>> wrote: Hi, You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid 😊 Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found". And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue. Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait.. Thank for your help, Frank ________________________________ From: Aaron Knister <[email protected]<mailto:[email protected]>> Sent: Saturday, June 10, 2017 2:05 PM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found Out of curiosity could you send us the output of "ibv_devinfo -v"? -Aaron Sent from my iPhone On Jun 10, 2017, at 06:55, Frank Tower <[email protected]<mailto:[email protected]>> wrote: Hi everybody, I don't get why one of our compute node cannot start GPFS over IB. I have the following error: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized. [I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)). [I] VERBS RDMA parse verbsPorts mlx4_0/1 [W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found [I] VERBS RDMA library libibverbs.so unloaded. [E] VERBS RDMA failed to start, no valid verbsPorts defined. I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64. I have 2 infinibands card, both have an IP and working well. [root@rdx110 ~]# ibstat -l mlx4_0 mlx4_1 [root@rdx110 ~]# I tried configuration with both card, and no one work with GPFS. I also tried with mlx4_0/1, but same problem. Someone already have the issue ? Kind Regards, Frank _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
