FYI the system is a supermicro based system with integrated ConnextX3 card
running 2.30....firmware info is below (obtained when installing
mlnx-en-2.3.1.0.0 from Mellanox website).
Device #1:
----------
Device Type: ConnectX3
Part Number: AOC-CIBF-M1
Description: ConnectX-3 single-port QSFP; FDR IB (56Gb/s) and
40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: SM_1101000001000
PCI Device Name: 0000:01:00.0
Port1 GUID: 002590ffff07f15d
Port2 MAC: 00259007f15e
Versions: Current Available
FW 2.30.3000 N/A
#ibv_devinfo -v
No IB devices found
lspci -vvv | grep -i Mellanox -A 40
01:00.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3]
Subsystem: Mellanox Technologies Device 0015
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at f7200000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at f5000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s
unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt-
ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not
Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+,
EqualizationPhase1+
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [148 v1] Device Serial Number 00-25-90-ff-ff-07-f1-5c
Capabilities: [108 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 63, Total VFs: 63, Number of VFs: 0, Function Dependency Link:
00
[root@l-cheddar1 fio]# ls /sys/class/net/enp1s0/device/ -l
total 0
.
.
-r--r--r--. 1 root root 4096 Oct 16 22:05 dma_mask_bits
lrwxrwxrwx. 1 root root 0 Oct 10 23:58 *driver ->
../../../../bus/pci/drivers/mlx4_core*
-rw-r--r--. 1 root root 4096 Oct 16 22:05 driver_override
-rw-r--r--. 1 root root 4096 Oct 16 22:05 enabled
lrwxrwxrwx. 1 root root 0 Oct 16 22:05 *firmware_node ->
../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:00/device:01*
-r-../../../../bus/pci
-r--r--r--. 1 root root 4096 Oct 16 22:05 subsystem_device
-r--r--r--. 1 root root 4096 Oct 16 22:05 subsystem_vendor
-rw-r--r--. 1 root root 4096 Oct 10 23:36 uevent
-r--r--r--. 1 root root 4096 Oct 10 23:37 vendor
[root@l-cheddar1 fio]#
On Thursday, October 16, 2014 11:04:40 AM UTC-7, Or Gerlitz wrote:
>
>
> On 10/16/2014 08:55 PM, Moussa Ba wrote:
>
> can ping the address, it seems to be going through the correct device.
>
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 0.0.0.0 10.102.40.1 0.0.0.0 UG 1024 0 0
> enp2s0f1
> 10.102.40.0 0.0.0.0 255.255.248.0 U 0 0 0
> enp2s0f1
> *192.168.111.0 0.0.0.0 255.255.255.0 U 0 0 0
> enp1s0*
> 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
> virbr0
>
>
> [jenkins@l-cheddar1 fio]$ sudo ip a s
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
> DOWN qlen 1000
> link/ether a0:36:9f:35:a5:00 brd ff:ff:ff:ff:ff:ff
> 3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> qlen 1000
> link/ether a0:36:9f:35:a5:02 brd ff:ff:ff:ff:ff:ff
> inet 10.102.43.86/21 brd 10.102.47.255 scope global enp2s0f1
> valid_lft forever preferred_lft forever
> inet6 fe80::a236:9fff:fe35:a502/64 scope link
> valid_lft forever preferred_lft forever
> 4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
> state DOWN
> link/ether fe:1d:0a:e1:9d:bb brd ff:ff:ff:ff:ff:ff
> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
> valid_lft forever preferred_lft forever
> *5: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> qlen 1000*
> link/ether 00:25:90:07:f1:5d brd ff:ff:ff:ff:ff:ff
> inet 192.168.111.101/24 brd 192.168.111.255 scope global enp1s0
> valid_lft forever preferred_lft forever
> inet6 fe80::225:90ff:fe07:f15d/64 scope link
> valid_lft forever preferred_lft forever
>
>
>
> does /sys/class/net/enp1s0 point to the ConnectX PCI device?
>
> does the mlx4_ib driver loaded?
>
> Can you please do the following
>
> $ yum install libmlx4
> $ yum install libibverbs
> $ yum install libibverbs-utils
> $ yum install rdma
>
> start the rdma service
>
> $ modprobe -v ib_uverbs
>
> $ ibv_devinfo -v
>
>
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/open-iscsi.
For more options, visit https://groups.google.com/d/optout.
