Hellp Gianluca,
Please find my responses inline ..
On 05/14/2010 11:40 AM, Gianluca Cecchi wrote:
Hello,
I found several guides (both Red Hat and Dell) about iscsi
configuration, but it seems to me some doubts remain about
configuration itself and functionality when using iscsi offload....
I can test two M610 blades, where I have RH EL 5.5 and RH EL 6 beta.
Both are x86_64 systems.
Below information from rh el 5.5 server
lspci gives:
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM57711 10-Gigabit PCIe
05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II
BCM57711 10-Gigabit PCIe
I was able to configure the two 10Gbit interfaces to connect to
several volumes on a Dell EQL 6010XV storage array, using sw based iscsi.
Also, I was able to configure dm-multipath so that each volume is seen
by two paths.
Some raw performance data show not more than 130-140MB/s (both by
using single devices and multipath devices) where other Windows based
blades, configured with Broadcom offload, reach more than 300MB/s on
the same storage array.
Trying to configure offload scsi on rh el I have these doubts:
My 10Gb BCM57711 card has two mac addresses (actually I have two of
these cards in total, to bind to the EQL).
00:10:18:58:E8:F8 base eth card for network
00:10:18:58:E8:F9 for iscsi offload (not seen from ifconfig, for example)
1) Do I have to configure an IP for both eth card and for the iscsi
offload mac? If so must they be different?
Or if I use offload, the nic is not assigned an ip at all?
If you use dhcp IP addressing then the following configuration should do
the trick..
# iscsiadm -m iface -I <iface name> -o update -n iface.ipaddress -v 0.0.0.0
where 0.0.0.0 will tell the adapter to get the ip for the offload interface.
But if you are using static addresses then you need to add them yourself
and different from the normal interface to avoid arp cache reference
problems..
2) Do the nic has to be configured anyway to start in
/etc/sysconfig/network-scripts/ifcfg-eth0?
With something like this for example if without an ip:
# Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe
DEVICE=eth0
HWADDR=00:10:18:58:E8:F8
ONBOOT=yes
BOOTPROTO=static
TYPE=Ethernet
MTU=9000
The discovery happens though the normal ethernet interface so you can
configrue the eth interface through the usual ways.
3) Can I set MTU=9000 if using offload scsi? Is it supported?
If I don't have instead to set the nic to start at boot, where to put
the mtu parameter?
That doesn't make a difference to the offload interface as this is only
for the eth interface.
4) I configured this way the iscsi offload iface, under
/var/lib/iscsi/ifaces dir:
# cat bnx2i.00:10:18:58:e8:f9
# BEGIN RECORD 2.0-871
iface.iscsi_ifacename = bnx2i.00:10:18:58:e8:f9
iface.ipaddress = 10.10.100.178
iface.hwaddress = 00:10:18:58:e8:f9
iface.transport_name = bnx2i
# END RECORD
and this for the nic associated:
[r...@orasvi2 ifaces]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:10:18:58:E8:F8
inet addr:10.10.100.174 Bcast:10.10.100.255 Mask:255.255.255.0
inet6 addr: fe80::210:18ff:fe58:e8f8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1683 errors:0 dropped:0 overruns:0 frame:0
TX packets:47 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:155976 (152.3 KiB) TX bytes:4562 (4.4 KiB)
Interrupt:114 Memory:dc800000-dcffffff
I have this results:
[r...@orasvi2 ifaces]# iscsiadm -m node -P1
Target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3
Portal: 10.10.100.30:3260 <http://10.10.100.30:3260>,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb
Target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1
Portal: 10.10.100.30:3260 <http://10.10.100.30:3260>,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb
Target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
Portal: 10.10.100.30:3260 <http://10.10.100.30:3260>,1
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Name: bnx2i.00:10:18:58:e8:fb
But actually I can connect only with one card and only to one
node..... ;-(
[r...@orasvi2 ifaces]# iscsiadm -m session -P1
Target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
Current Portal: 10.10.100.32:48140 <http://10.10.100.32:48140>,1
Persistent Portal: 10.10.100.30:3260 <http://10.10.100.30:3260>,1
**********
Interface:
**********
Iface Name: bnx2i.00:10:18:58:e8:f9
Iface Transport: bnx2i
Iface Initiatorname: iqn.1994-05.com.redhat:ab59ecbce6c
Iface IPaddress: <empty>
Iface HWaddress: 00:10:18:58:e8:f9
Iface Netdev: eth0
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
Note the
Iface IPaddress: <empty> ...
Just a basic test shows, for this single out of three volumes:
dd if=/dev/sdb of=/dev/null bs=10240k count=9500
[r...@orasvi2 ~]# vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us
sy id wa st
1 1 24 139100 46577740 43056 0 0 3513 2 203 533 0
0 97 2 0
2 0 24 139660 46576220 42552 0 0 108117 23 3196 6280
0 2 87 11 0
1 1 24 140576 46575508 42676 0 0 120064 0 3496 6508
0 2 87 11 0
1 1 24 137440 46580264 42196 0 0 117888 0 3410 6429
0 2 87 11 0
2 0 24 141512 46577092 41888 0 0 134741 17 3956 6979
0 2 87 11 0
1 1 24 138568 46580280 41944 0 0 117376 0 3508 6458
0 2 87 11 0
0 1 24 137620 46581736 41404 0 0 124629 0 3740 6088
0 2 87 11 0
0 1 24 137148 46582524 41144 0 0 116651 11 3512 5570
0 2 87 11 0
0 1 24 141428 46578336 40524 0 0 130560 32 3879 6844
0 2 87 11 0
0 1 24 145376 46574348 41268 0 0 118997 0 3590 6512
0 2 87 11 0
0 1 24 136468 46583624 40696 0 0 119467 0 3590 5904
0 2 87 11 0
Logs I can see:
May 14 16:35:10 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:10 orasvi2 iscsid: Received iferror -19
May 14 16:35:10 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 <http://10.10.100.30:3260> (-19,11)
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:11 orasvi2 iscsid: Received iferror -101
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 <http://10.10.100.30:3260> (-101,11)
May 14 16:35:11 orasvi2 iscsid: Received iferror -19
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 <http://10.10.100.30:3260> (-19,11)
May 14 16:35:11 orasvi2 iscsid: Received iferror -101
May 14 16:35:11 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 <http://10.10.100.30:3260> (-101,11)
May 14 16:35:12 orasvi2 kernel: bnx2i [05:00.01]: ISCSI_INIT passed
May 14 16:35:12 orasvi2 iscsid: Received iferror -101
May 14 16:35:12 orasvi2 iscsid: cannot make a connection to
10.10.100.30:3260 <http://10.10.100.30:3260> (-101,11)
May 14 16:35:13 orasvi2 kernel: connection1:0: detected conn error (1011)
May 14 16:35:13 orasvi2 iscsid: Login authentication failed with
target
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2
May 14 16:35:15 orasvi2 kernel: bnx2i [05:00.00]: ISCSI_INIT passed
May 14 16:35:17 orasvi2 kernel: connection1:0: bnx2i: conn update -
MBL 0x40000 FBL 0x10000MRDSL_I 0x40000 MRDSL_T 0x10000
May 14 16:35:17 orasvi2 kernel: Vendor: EQLOGIC Model: 100E-00
Rev: 4.3
May 14 16:35:17 orasvi2 kernel: Type: Direct-Access
ANSI SCSI revision: 05
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: 419450880 512-byte
hdwr sectors (214759 MB)
May 14 16:35:17 orasvi2 kernel: sdb: Write Protect is off
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: drive cache: write
through
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: 419450880 512-byte
hdwr sectors (214759 MB)
May 14 16:35:17 orasvi2 kernel: sdb: Write Protect is off
May 14 16:35:17 orasvi2 kernel: SCSI device sdb: drive cache: write
through
May 14 16:35:17 orasvi2 kernel: sdb: unknown partition table
May 14 16:35:17 orasvi2 kernel: sd 2:0:0:0: Attached scsi disk sdb
May 14 16:35:17 orasvi2 multipathd: sdb: add path (uevent)
May 14 16:35:17 orasvi2 kernel: sd 2:0:0:0: Attached scsi generic sg3
type 0
May 14 16:35:17 orasvi2 kernel: device-mapper: multipath round-robin:
version 1.0.0 loaded
May 14 16:35:17 orasvi2 multipathd: vol2: load table [0 419450880
multipath 1 queue_if_no_path 0 1 1 round-robin 0 1 1 8:16 10]
May 14 16:35:17 orasvi2 multipathd: vol2: event checker started
May 14 16:35:17 orasvi2 multipathd: dm-3: add map (uevent)
May 14 16:35:17 orasvi2 multipathd: dm-3: devmap already registered
May 14 16:35:17 orasvi2 iscsid: connection1:0 is operational now
May 14 16:35:17 orasvi2 iscsid: Could not write to
/sys/bus/scsi/devices/2:0:0:0/queue_depth. Invalid permissions.
May 14 16:35:17 orasvi2 iscsid: Could not queue depth for LUN 0 err 13.
and
# multipath -l
vol2 (36090a068e0b5c49483bc3402000090df) dm-3 EQLOGIC,100E-00
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 2:0:0:0 sdb 8:16 [active][undef]
If I now try to connect with the other card ending fb mac .175 ip for
eth1 and .179 for the iscsi part):
# iscsiadm --mode node --portal 10.10.100.30:3260
<http://10.10.100.30:3260> -I bnx2i.00:10:18:58:e8:fb --login
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3,
portal: 10.10.100.30,3260]
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1,
portal: 10.10.100.30,3260]
Logging in to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2,
portal: 10.10.100.30,3260]
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-97d4b5e06-596000000264bc83-blg9-vol3,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-8904b5e06-b66000000204bc83-blg9-vol1,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: bnx2i.00:10:18:58:e8:fb, target:
iqn.2001-05.com.equallogic:0-8a0906-94c4b5e06-df9000000234bc83-blg9-vol2,
portal: 10.10.100.30,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
It is a good idea to discover for the other iface as well to create a
new node record for the interface
#iscsiadm -m discovery -t st -p 10.10.100.30:3260
<http://10.10.100.30:3260> -I bnx2i.00:10:18:58:e8:fb
and then login
# iscsiadm --mode node --portal 10.10.100.30:3260
<http://10.10.100.30:3260> -I bnx2i.00:10:18:58:e8:fb --login
-Shyam Iyer
Dell Linux Engineering
_______________________________________________
Linux-PowerEdge mailing list
[email protected]
https://lists.us.dell.com/mailman/listinfo/linux-poweredge
Please read the FAQ at http://lists.us.dell.com/faq
