[PVE-User] Infiniband Mellanox cards and DDR
Hallo. I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch. The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link. --- 03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20) --- root@pve1:/etc/apt/sources.list.d# ibstat CA 'mthca0' CA type: MT25208 Number of ports: 2 Firmware version: 5.3.0 Hardware version: 20 Node GUID: 0x0008f104039814d0 System image GUID: 0x0008f104039814d3 Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 6 LMC: 0 SM lid: 2 Capability mask: 0x02510a68 Port GUID: 0x0008f104039814d1 Link layer: InfiniBand root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 - If I try to set the speed at DDR, I have this error: root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2 Initial CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 ibportstate: iberror: failed: smp set portinfo failed The switch reported this card in infiniband topology: MT25218 InfiniHostEx Mellanox Technologies Someone have tested the cards ? Fabrizio Cuseo -- --- Fabrizio Cuseo - mailto:f.cu...@panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:i...@panservice.it Numero verde nazionale: 800 901492 ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Infiniband Mellanox cards and DDR
Another thing is that your cables must support DDR as well. On July 7, 2015 3:07:00 PM CEST, Fabrizio Cuseo f.cu...@panservice.it wrote: I have found that the 400EX is a SDR card, while the 400EX-D is a DDR card. I don't know if is a firmware problem. http://sup.xenya.si/sup/info/voltaire/HCAwebfinal.pdf HCA 400Ex/Ex-F • Dual port 4X (10 Gbps) InfiniBand PCI-Express low profile host channel adapter HCA 400Ex-D • Dual port 4X DDR (20 Gbps) InfiniBand PCI-Express low profile host channel adapter HCA 400 • Dual port 4X (10 Gbps) InfiniBand PCI/PCI-X low profile host channel adapter - Messaggio originale - Da: Michael Rasmussen m...@miras.org A: Fabrizio Cuseo f.cu...@panservice.it, pve-user pve-user@pve.proxmox.com Inviato: Martedì, 7 luglio 2015 12:01:03 Oggetto: Re: [PVE-User] Infiniband Mellanox cards and DDR AFAIK this card is a SDR card so DDR is not possible. On July 7, 2015 11:46:41 AM CEST, Fabrizio Cuseo f.cu...@panservice.it wrote: Hallo. I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch. The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link. --- 03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20) --- root@pve1:/etc/apt/sources.list.d# ibstat CA 'mthca0' CA type: MT25208 Number of ports: 2 Firmware version: 5.3.0 Hardware version: 20 Node GUID: 0x0008f104039814d0 System image GUID: 0x0008f104039814d3 Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 6 LMC: 0 SM lid: 2 Capability mask: 0x02510a68 Port GUID: 0x0008f104039814d1 Link laye r: InfiniBand root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 - If I try to set the speed at DDR, I have this error: root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2 Initial CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 ibportstate: iberror: failed: smp set portinfo failed The switch reported this card in infiniband topology: MT25218 InfiniHostEx Mellanox Technologies Someone have tested the cards ? Fabrizio Cuseo -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. This mail was virus scanned and spam checked before delivery. This mail is also DKIM signed. See header dkim-signature. -- --- Fabrizio Cuseo - mailto:f.cu...@panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:i...@panservice.it Numero verde nazionale: 800 901492 -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. This mail was virus scanned and spam checked before delivery. This mail is also DKIM signed. See header dkim-signature. ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Infiniband Mellanox cards and DDR
I have found that the 400EX is a SDR card, while the 400EX-D is a DDR card. I don't know if is a firmware problem. http://sup.xenya.si/sup/info/voltaire/HCAwebfinal.pdf HCA 400Ex/Ex-F • Dual port 4X (10 Gbps) InfiniBand PCI-Express low profile host channel adapter HCA 400Ex-D • Dual port 4X DDR (20 Gbps) InfiniBand PCI-Express low profile host channel adapter HCA 400 • Dual port 4X (10 Gbps) InfiniBand PCI/PCI-X low profile host channel adapter - Messaggio originale - Da: Michael Rasmussen m...@miras.org A: Fabrizio Cuseo f.cu...@panservice.it, pve-user pve-user@pve.proxmox.com Inviato: Martedì, 7 luglio 2015 12:01:03 Oggetto: Re: [PVE-User] Infiniband Mellanox cards and DDR AFAIK this card is a SDR card so DDR is not possible. On July 7, 2015 11:46:41 AM CEST, Fabrizio Cuseo f.cu...@panservice.it wrote: Hallo. I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch. The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link. --- 03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20) --- root@pve1:/etc/apt/sources.list.d# ibstat CA 'mthca0' CA type: MT25208 Number of ports: 2 Firmware version: 5.3.0 Hardware version: 20 Node GUID: 0x0008f104039814d0 System image GUID: 0x0008f104039814d3 Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 6 LMC: 0 SM lid: 2 Capability mask: 0x02510a68 Port GUID: 0x0008f104039814d1 Link laye r: InfiniBand root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 - If I try to set the speed at DDR, I have this error: root@pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2 Initial CA PortInfo: # Port info: Lid 6 port 0 LinkState:...Active PhysLinkState:...LinkUp Lid:.6 SMLid:...2 LMC:.0 LinkWidthSupported:..1X or 4X LinkWidthEnabled:1X or 4X LinkWidthActive:.4X LinkSpeedSupported:..2.5 Gbps LinkSpeedEnabled:2.5 Gbps LinkSpeedActive:.2.5 Gbps Mkey:not displayed MkeyLeasePeriod:.15 ProtectBits:.0 ibportstate: iberror: failed: smp set portinfo failed The switch reported this card in infiniband topology: MT25218 InfiniHostEx Mellanox Technologies Someone have tested the cards ? Fabrizio Cuseo -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. This mail was virus scanned and spam checked before delivery. This mail is also DKIM signed. See header dkim-signature. -- --- Fabrizio Cuseo - mailto:f.cu...@panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:i...@panservice.it Numero verde nazionale: 800 901492 ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
