I'm experiencing very bad perfomance for IPOIB (ip over Infiniband) on
opensolaris b77.
Setup was easy, but now I'm strugling with the resulted perfomance.
I have 2 opensolaris boxes, each have a dual port Voltaire HCA400-EX card.
When I measure perfomance with iperf I get really bad results. (ports are
listed as
Running 1 parrallel test I get +/- 900 Mb/sec, which is less what I see with
Giga Ethernet (940 Mb/s)
[EMAIL PROTECTED]:53 PM-~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 63388
[ 4] 0.0-10.0 sec 1.08 GBytes 929 Mbits/sec
[ 5] local 192.168.4.253 port 5001 connected with 192.168.4.252 port 59380
[ 5] 0.0-10.0 sec 1.01 GBytes 872 Mbits/sec
When I run iperf with 8 parrallel threads, perfomance drops to +/- 530 Mb/sec
[EMAIL PROTECTED]:54 PM-~]# iperf -s -P 8
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 53328
[ 5] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 43744
[ 6] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 44023
[ 7] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 57248
[ 8] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 42721
[ 9] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 60192
[ 10] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 41659
[ 11] local 192.168.3.253 port 5001 connected with 192.168.3.252 port 42606
[ 11] 0.0-10.0 sec 125 MBytes 105 Mbits/sec
[ 6] 0.0-10.0 sec 125 MBytes 105 Mbits/sec
[ 10] 0.0-10.0 sec 125 MBytes 105 Mbits/sec
[ 8] 0.0-10.0 sec 125 MBytes 105 Mbits/sec
[ 9] 0.0-10.0 sec 33.9 MBytes 28.4 Mbits/sec
[ 5] 0.0-10.0 sec 33.7 MBytes 28.2 Mbits/sec
[ 7] 0.0-10.0 sec 33.7 MBytes 28.3 Mbits/sec
[ 4] 0.0-10.0 sec 33.5 MBytes 28.1 Mbits/sec
[SUM] 0.0-10.0 sec 635 MBytes 532 Mbits/sec
I found the following messages in the logs:
[EMAIL PROTECTED]:00 PM-~]# grep tavor /var/adm/messages
Jan 2 10:37:55 firewall pcplusmp: [ID 803547 kern.info] pcplusmp:
pciex15b3,6278 (tavor) instance 0 vector 0x58 ioapic 0xff intin 0xff is bound
to cpu 5
Jan 2 10:37:55 firewall pcie_pci: [ID 586369 kern.info] PCIE-device:
pci15b3,[EMAIL PROTECTED], tavor0
Jan 2 10:37:55 firewall genunix: [ID 936769 kern.info] tavor0 is /[EMAIL
PROTECTED],0/pci8086,[EMAIL PROTECTED]/pci15b3,[EMAIL PROTECTED]
Jan 2 10:37:55 firewall tavor: [ID 237812 kern.info] tavor0: FW ver:
0004.0007.0600, HW rev: a0
Jan 2 10:37:55 firewall tavor: [ID 124060 kern.info] tavor0:
Voltaire HCA400-EX (0x0008f10403988f14)
Jan 2 10:37:55 firewall genunix: [ID 408822 kern.info] NOTICE: tavor0: no
fault external to device; service available
Jan 2 10:37:55 firewall genunix: [ID 611667 kern.info] NOTICE: tavor0: port 1
up
Jan 2 10:37:55 firewall genunix: [ID 408822 kern.info] NOTICE: tavor0: no
fault external to device; service available
Jan 2 10:37:55 firewall genunix: [ID 611667 kern.info] NOTICE: tavor0: port 2
up
Jan 2 10:37:57 firewall genunix: [ID 408114 kern.info] /[EMAIL
PROTECTED],0/pci8086,[EMAIL PROTECTED]/pci15b3,[EMAIL PROTECTED] (tavor0) online
Jan 2 10:38:06 firewall genunix: [ID 834635 kern.info] /ib/[EMAIL PROTECTED]
(daplt0) multipath status: degraded, path /[EMAIL PROTECTED],0/pci8086,[EMAIL
PROTECTED]/pci15b3,[EMAIL PROTECTED] (tavor0) to target address: daplt,0 is
online Load balancing: round-robin
Jan 2 10:38:06 firewall genunix: [ID 834635 kern.info] /ib/[EMAIL PROTECTED]
(rpcib0) multipath status: degraded, path /[EMAIL PROTECTED],0/pci8086,[EMAIL
PROTECTED]/pci15b3,[EMAIL PROTECTED] (tavor0) to target address: rpcib,0 is
online Load balancing: round-robin
Jan 2 10:38:06 firewall genunix: [ID 834635 kern.info] /ib/[EMAIL PROTECTED]
(rdsib0) multipath status: degraded, path /[EMAIL PROTECTED],0/pci8086,[EMAIL
PROTECTED]/pci15b3,[EMAIL PROTECTED] (tavor0) to target address: rdsib,0 is
online Load balancing: round-robin
Jan 2 10:38:06 firewall genunix: [ID 834635 kern.info] /ib/[EMAIL PROTECTED]
(sdpib0) multipath status: degraded, path /[EMAIL PROTECTED],0/pci8086,[EMAIL
PROTECTED]/pci15b3,[EMAIL PROTECTED] (tavor0) to target address: sdpib,0 is
online Load balancing: round-robin
dladm shows the following output:
server1:
[EMAIL PROTECTED]:00 PM-~]# dladm show-link
bnx1 type: legacy mtu: 1500 device: bnx1
e1000g0 type: non-vlan mtu: 1500 device: e1000g0
e1000g1 type: non-vlan mtu: 1500 device: e1000g1
e1000g2 type: non-vlan mtu: 1500 device: e1000g2
e1000g3 type: non-vlan mtu: 1500 device: e1000g3
ibd0 type: legacy mtu: 2044 device: ibd0
ibd1 type: legacy mtu: 2044 device: ibd1
bnx0 type: legacy mtu: 1500 device: bnx0
aggr1 type: non-vlan mtu: 1500 aggregation: key 1
aggr2 type: non-vlan mtu: 1500 aggregation: key 2
aggr1001001 type: vlan 1001 mtu: 1500 aggregation: key 1
aggr1002001 type: vlan 1002 mtu: 1500 aggregation: key 1
[EMAIL PROTECTED]:02 PM-~]#
bnx1 link: unknown speed: 0Mb duplex: unknown
e1000g0 link: up speed: 1000Mb duplex: full
e1000g1 link: up speed: 1000Mb duplex: full
e1000g2 link: up speed: 1000Mb duplex: full
e1000g3 link: up speed: 1000Mb duplex: full
ibd0 link: unknown speed: 8000Mb duplex: unknown
ibd1 link: unknown speed: 8000Mb duplex: unknown
bnx0 link: unknown speed: 0Mb duplex: unknown
server 2
[EMAIL PROTECTED]:02 PM-~]# dladm show-link
bnx1 type: legacy mtu: 1500 device: bnx1
e1000g0 type: non-vlan mtu: 1500 device: e1000g0
e1000g1 type: non-vlan mtu: 1500 device: e1000g1
e1000g2 type: non-vlan mtu: 1500 device: e1000g2
e1000g3 type: non-vlan mtu: 1500 device: e1000g3
ibd0 type: legacy mtu: 2044 device: ibd0
ibd1 type: legacy mtu: 2044 device: ibd1
bnx0 type: legacy mtu: 1500 device: bnx0
aggr1 type: non-vlan mtu: 1500 aggregation: key 1
aggr2 type: non-vlan mtu: 1500 aggregation: key 2
aggr1001001 type: vlan 1001 mtu: 1500 aggregation: key 1
aggr1002001 type: vlan 1002 mtu: 1500 aggregation: key 1
aggr1005001 type: vlan 1005 mtu: 1500 aggregation: key 1
bnx1 link: unknown speed: 0Mb duplex: unknown
e1000g0 link: up speed: 1000Mb duplex: full
e1000g1 link: up speed: 1000Mb duplex: full
e1000g2 link: up speed: 1000Mb duplex: full
e1000g3 link: up speed: 1000Mb duplex: full
ibd0 link: unknown speed: 8000Mb duplex: unknown
ibd1 link: unknown speed: 8000Mb duplex: unknown
bnx0 link: unknown speed: 0Mb duplex: unknown
And finaly the ifconfig output:
ibd0: flags=201100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4,CoS> mtu 2044
index 6
inet 192.168.4.252 netmask ffffff00 broadcast 192.168.4.255
ipib 0:0:4:8:fe:80:0:0:0:0:0:0:0:8:f1:4:3:98:90:29
[EMAIL PROTECTED]:08 PM-~]# ifconfig ibd1
ibd1: flags=201100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4,CoS> mtu 2044
index 7
inet 192.168.3.252 netmask ffffff00 broadcast 192.168.3.255
ipib 0:0:4:9:fe:80:0:0:0:0:0:0:0:8:f1:4:3:98:90:2a
ibd0: flags=201100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4,CoS> mtu 2044
index 7
inet 192.168.4.253 netmask ffffff00 broadcast 192.168.4.255
ipib 0:0:4:8:fe:80:0:0:0:0:0:0:0:8:f1:4:3:98:8f:15
[EMAIL PROTECTED]:08 PM-~]# ifconfig ibd1
ibd1: flags=201100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4,CoS> mtu 2044
index 8
inet 192.168.3.253 netmask ffffff00 broadcast 192.168.3.255
ipib 0:0:4:9:fe:80:0:0:0:0:0:0:0:8:f1:4:3:98:8f:16
Any body any idea how to tune IB on opensolaris ??
PS: We are using a voltaire IB switch which supports 4X, and it has an
integrated subnet manager.
thanks for your reply.
K
This message posted from opensolaris.org
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
