We have created the detailed procedure for FCoE Initiator and FCoE Target
setup document. We have also listed the problems faced during setup and its
solutions.
Please feel free to provide your comments.
=============================
FCoE Target Setup Procedure
=============================
Note: Perform FCoE Target setup first. Then setup the FCoE Initiator.
[If the Initiator is setup first, the Target side could crash.
For proper SCSI Target initialization, FLOGI command should be initiated by
FCoE Initiator and accepted by FCoE Target; which brings the Target to Ready
state.]
Packages Required:
------------------
1. Download the kernel v2.6.23
a) Kernel is located in http://www.kernel.org/pub/linux/kernel/v2.6/
2. Download open-fcoe-target and open-fcoe-target-user
a) It is present in
http://www.open-fcoe.org/wordpress/frame.php?url=http://www.open-fcoe.org/op
enfc/gitweb/
b) Click summary link of open-fcoe-target and open-fcoe-target-user.
c) Click snapshot link of open-fcoe-target (master) and
open-fcoe-target-user (master) to download.
3. Download SCSI Target and its patch
a) SCST is present in
http://scst.svn.sourceforge.net/viewvc/scst/trunk/scst/
4. Download HBAAPI source ("hbaapi_src_2.2.gz")
a) It is present in http://sourceforge.net/projects/hbaapi/
5. Download HBAAPI Patch (hbaapi2.2.patch)
a) It is present in
http://www.open-fcoe.org/wordpress/frame.php?url=http://www.open-fcoe.org/op
enfc/gitweb/
b) Click summary link of hbaapi_build.
c) Click snapshot link to download hbaapi_build.
d) Untar the hbaapi_build.tar.gz and copy "hbaapi2.2.patch" file
into the system.
6. Download lsscsi package ("lsscsi-0.19.tgz0")
a) It is present in http://sg.danny.cz/scsi/lsscsi.html
Initial Setup:
---------------
1. Kernel build
a) cd <path-to-kernel-source>
b) make menuconfig, exit saving changes
c) make bzImage
d) make modules
e) make modules_install
f) mkinitrd /boot/initrd-2.6.23.img
g) cp arch/i386/boot/bzImage /boot/bzImage-2.6.23
h) System.map /boot/System.map-2.6.23
i) ln -s /boot/System.map-2.6.23 /boot/System.map
j) Update the new kernel entry in grub/lilo bootloader file and
reboot the system into new kernel.
Target Modules Build:
---------------------
1. Build SCST
a) cd <path-to-scst>/trunk/scst/src
b) Open the Makefile and Check CONFIG_SCST_STRICT_SERIALIZING flag
is defined. If it is not, define it. (EXTRA_CFLAGS += -
DCONFIG_SCST_STRICT_SERIALIZING)
c) make && make install
2. Build openfctgt and fcoe
a) cd <path-to-open-fcoe-target>
b) make SCST=<path-to-scst>/trunk/scst install_tgt
3. Build the HBA API
a) cd <path-to-HBAAPI-source>
b) Apply the hbaapi2.2.patch to HBAAPILIB.c using the command "patch
HBAAPILIB.c < <path-to-hbaapi-Patch>/hbaapi2.2.patch"
c) cc -m32 -c HBAAPILIB.c -fPIC -DVENDOR='"Your Co. Here, Inc."' \
> -DREVISION='"Rev 2.2"' -DREVNUM=2 -DMINREVNUM=2 \
> -DLICENSE='"Subject to SNIA Public License"' \
> -DBUILD_DATE="\"`date "+%m/%d/%y %H:%M:%S"`\""
d) ld -melf_i386 HBAAPILIB.o -G -o libHBAAPI.so
e) install libHBAAPI.so /usr/lib
]# cc -c HBAAPILIB.c -fPIC -DVENDOR='"Your Co. Here, Inc."' \
> -DREVISION='"Rev 2.2"' -DREVNUM=2 -DMINREVNUM=2 \
> -DLICENSE='"Subject to SNIA Public License"' \
> -DBUILD_DATE="\"`date "+%m/%d/%y %H:%M:%S"`\""
f) ld HBAAPILIB.o -G -o libHBAAPI.so
g) install libHBAAPI.so /usr/lib
h) ldconfig
4) Build fcconf
a) cd <path-to-open-fcoe-target-user>/usr
b) make HBAAPI=<absolute-path-to-hbaapi-src> && make install
5) Build lsscsi
a)./configure
b) make
c) make install
Loading SCSI Target Modules and its supported modules:
------------------------------------------------------
1. Load modules in the order given below
a) modprobe scsi_trasport_fc
b) modprobe scst
c) modprobe scst_disk (In case block disk to be added)
d) modprobe scst_vdisk (In case virtual disk to be added)
e) modprobe scst_cdrom (In case cdrom to be added)
f) modprobe openfctgt
g) modprobe fcoe
Note: In case loading of fcoe fails, try loading using insmod as given
below.
. insmod <path-to-open-target>/openfctgt.ko
. insmod <path-to-open-target>/fcoe.ko
2. Verify that all the modules are loaded successfully by using lsmod
command.
Exporting SCSI Target Devices:
-------------------------------
1. Get the List of available SCSI devices by using lsscsi command.
[The list below is specific to our machine]
[r...@fcoe ~]# lsscsi
[0:0:0:0] disk ATA ST380815AS 3.CH /dev/sda
[1:0:0:0] cd/dvd ATAPI DVD D DH16DYS XH31 /dev/scd0
[r...@fcoe ~]#
2. Add the available disks to SCST by using below commands.
a) Echo "assign 0:0:0:0 dev_disk" > /proc/scsi_tgt/scsi_tgt
b) Echo "add 0:0:0:0 0" > /proc/scsi_tgt/groups/Default/devices
3. Verify that disks are properly added to SCST by using below command.
[r...@fcoe ~]# cat /proc/scsi_tgt/groups/Default/devices
Device (host:ch:id:lun or name) LUN
Options
0:0:0:0 0
[r...@fcoe ~]#
4. Verify that transmission and reception of PAUSE are set to ON by using
below command.
[r...@fcoe ~]# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate: on
RX: on
TX: on
5. Start the target by using below command
a) fcconf ethX enable - where ethX is your device connected to
Initiator.
================================
FCoE Initiator Setup Procedure:
================================
Packages Required:
-------------------
1. Download the kernel in which open-fcoe is supported
a) FCoE is supported in v2.6.29 and above kernels
b) Kernels are located in
http://www.kernel.org/pub/linux/kernel/v2.6/
Initial Setup:
--------------
1. Kernel build
a) cd <path-to-kernel-source>
b) make menuconfig
Select -> Device drivers-> SCSI device support-> SCSI Transports
Enable modularizes features for Fiber channel Transport
Attributes
Select -> Device drivers-> SCSI device support-> SCSI Low level
drivers
Enable modularizes features for LibFC module, FCoE module
c) make modules
d) make modules_install
e) mkinitrd /boot/<image-name>
f) cp arch/i386/boot/bzImage /boot/bzImage-KERNEL_VERSION
g) cp System.map /boot/System.map-KERNEL_VERSION
h) ln -s /boot/System.map-KERNEL_VERSION /boot/System.map
i) Update the new kernel entry in grub or lilo bootloader file and
reboot the system in new kernel.
Initiator Setup:
----------------
1. Setup FCoE Target
2. Load fcoe module using -
a) modprobe fcoe
3. Start the initiator on ethernet interface ethX (where ethX is the device
connected to the target)
a) echo "ethX" > /sys/module/fcoe/parameters/create
(Execution of the above command will create scsi_host and fc_host
entries in /sys/class directory)
Note: The FCoE Target setup should be done before executing the
above command.
4. Verify that new SCSI host and FC host are created by using below command.
a) ls /sys/class/scsi_host/hostX
b) ls /sys/class/fc_host/hostX
where X is the newly created Host number.
4. Run wireshark to verify FCoE traffic.
5. Verify that exported devices from Target are available in Initiator by
using below command
a) cat /proc/scsi/scsi
- Newly detected devices will be present under Host number created
by FCoE.
b) cat /proc/partition
- Newly detected device will be present.
c) ls /sys/block/sdX/device/
- Mapping of Device name and H C T L can be verified here.
7. If the target device is block device, then Create a filesystem on the
detected device and mount the filesystem and perform basic read/write
operation on the device. Confirm read/write operation is successful.
Destroy the Initiator:
----------------------
1. Execute the below command to remove the FCoE Interface.
a. echo "ethX" > /sys/module/fcoe/parameters/destroy
(Execution of the above command will destroy scsi_host and fc_host
entries in /sys/class directory)
2. unload the fcoe module
a) rmmod fcoe
=================================================
Issues observed during Setup and Its solutions:
=================================================
#1. [Issue]
-----------
During building of SCST module, the error message related with "page member
in scatterlist structure" is logged in syslog, when kernel 2.6.29 is used.
[Solution]
For FCoE Target setup, 2.6.23 kernel must be used. Using other kernels leads
to such problem.
(The scatterlist structure in 2.6.29 kernel is updated; page member has been
removed. Hence the above problem occurs).
#2. [Issue]
-----------
During loading of scst_disk is fails with following error message.
[r...@fcoe ~]# modprobe scst_disk
FATAL: Error inserting scst_disk
(/lib/modules/2.6.23/extra/dev_handlers/scst_disk.ko): Invalid argument
[r...@fcoe ~]#
Error messages observed in syslog:
Jul 23 14:40:28 FCoE kernel: [6113]: scst: scst_print_config:1673:Enabled
features: EXTRACHECKS, DEBUG
Jul 23 14:40:28 FCoE kernel: [6126]: scst:
scst_global_mgmt_thread:5582:Management thread started, PID 6126
Jul 23 14:40:30 FCoE kernel: [6179]: scst:
__scst_register_dev_driver:930:***ERROR*** Pass-through dev handlers
(handler "dev_disk") not supported. Consider applying on your kernel patch
scst_exec_req_fifo-<kernel-version> or define CONFIG_SCST_STRICT_SERIALIZING
Jul 23 14:40:30 FCoE kernel: [6179]: scst:
__scst_register_dev_driver:991:***ERROR*** Failed to register device handler
"dev_disk" for type 0
[Solution]
1. cd <path-to-scst>/trunk/scst/src
2. Open the Makefile and Check CONFIG_SCST_STRICT_SERIALIZING flag is
defined. If it is not, define it. (EXTRA_CFLAGS +=
-DCONFIG_SCST_STRICT_SERIALIZING)
3. Build SCST modules.
4. Build openfctgt and fcoe modules.
#3. [Issue]
-----------
Loading of openfctgt.ko module fails with following error message.
[r...@fcoe ~]# insmod /home/FCoE_Target/open-fcoe-target/openfctgt.ko
insmod: error inserting '/home/FCoE_Target/open-fcoe-target/openfctgt.ko':
-1 Invalid module format
[r...@fcoe ~]#
Error message observed in syslog:
Jul 14 14:58:14 FCoE kernel: openfc_ioctl_init: Control device /dev/ofc
major number 253
Jul 14 14:58:41 FCoE kernel: openfctgt: exports duplicate symbol openfc_rcv
(owned by openfc)
[Solution]
1. Check whether openfc module is loaded by using lsmod command.
2. If yes, remove the module by using "rmmod openfc" command.
3. Load openfctgt.ko module.
#4. [Issue]
-----------
Execution of "fcconf eth0 enable" command fails with following error.
[r...@fcoe ~]# fcconf eth0 enable
fcconf: fcc_fcoe_config: FCoE create of eth0 failed
fcconf: fcc_fcoe_config: error 95 Operation not supported
fcconf: fcc_fcoe_config: exiting at fcc_fcoe.c:142
[r...@fcoe ~]#
[Solution]
Transmission and Reception of PAUSE are not set to ON. It can be set to ON
by using the command "ethtool -A tx on rx on"
#5. [Issue]
------------
Execution of "fcconf eth0 enable" command hangs and below call trace is
logged in syslog.
Jul 3 12:48:55 FCoE kernel: ------------[ cut here ]------------
Jul 3 12:48:55 FCoE kernel: kernel BUG at
/home/FCoE_Target/open-fcoe-target/openfctgt/openfc_scst_pkt.h:119!
Jul 3 12:48:55 FCoE kernel: invalid opcode: 0000 [#1]
Jul 3 12:48:55 FCoE kernel: SMP
Jul 3 12:48:55 FCoE kernel: Modules linked in: scst_cdrom fcoe openfctgt
scst_vdisk scst_disk scst scsi_transport_fc parport_pc lp parport autofs4
sunrpc dm_mirror dm_mod button battery ac ipv6 ohci_hcd ehci_hcd i2c_piix4
i2c_core tg3 floppy sr_mod ext3 jbd ahci libata sd_mod scsi_mod
Jul 3 12:48:55 FCoE kernel: CPU: 1
Jul 3 12:48:55 FCoE kernel: EIP: 0060:[<f8ae4027>] Not tainted VLI
Jul 3 12:48:55 FCoE kernel: EFLAGS: 00010246 (2.6.23 #1)
Jul 3 12:48:55 FCoE kernel: EIP is at openfct_session_create+0x18/0xd1
[openfctgt]
Jul 3 12:48:55 FCoE kernel: eax: d58f8000 ebx: eb2bda80 ecx: f8ae400f
edx: eb2bda80
Jul 3 12:48:55 FCoE kernel: esi: 00010102 edi: 00000000 ebp: 00000000
esp: d1835e24
Jul 3 12:48:55 FCoE kernel: ds: 007b es: 007b fs: 00d8 gs: 0000 ss:
0068
Jul 3 12:48:55 FCoE kernel: Process fcoethread/1 (pid: 20832, ti=d1835000
task=df9d9020 task.ti=d1835000)
Jul 3 12:48:55 FCoE kernel: Stack: c02ba21b ccf0f580 c8a00710 f8ae6be9
d89d0840 c18c9f0c 00000000 c18c9f08
Jul 3 12:48:55 FCoE kernel: d1835ee0 c01df0f8 00000001 c0121117
ccf0fe40 eb2bda80 e146be00 00000000
Jul 3 12:48:55 FCoE kernel: f8ae6002 c8a00810 c031fe31 f8ae6ea3
00000001 f8ae5fb0 d7b91f24 e146be00
Jul 3 12:48:55 FCoE kernel: Call Trace:
Jul 3 12:48:55 FCoE kernel: [<c02ba21b>] __alloc_skb+0x25/0xef
Jul 3 12:48:55 FCoE kernel: [<f8ae6be9>] fc_exch_complete_locked+0x6d/0x9b
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<c01df0f8>] __next_cpu+0x12/0x1f
Jul 3 12:48:55 FCoE kernel: [<c0121117>] find_busiest_group+0x1bc/0x505
Jul 3 12:48:55 FCoE kernel: [<f8ae6002>]
fcs_local_port_prli_accept+0x52/0x95 [openfctgt]
Jul 3 12:48:55 FCoE kernel: [<c031fe31>] _spin_unlock_bh+0x5/0xd
Jul 3 12:48:55 FCoE kernel: [<f8ae6ea3>] fc_exch_alloc+0x26e/0x27a
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8ae5fb0>]
fcs_local_port_prli_accept+0x0/0x95 [openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8af2005>] fc_sess_recv_prli_req+0xa8/0x5b1
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8af29dc>] fc_sess_lookup_create+0x103/0x10d
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8ae6ecc>] fc_exch_resp+0xc/0x150
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8aedca0>] fc_local_port_recv_req+0xb1/0x15b
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8ae8557>] fc_exch_recv_req+0x396/0x457
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8aedbef>] fc_local_port_recv_req+0x0/0x15b
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8aedec9>] fc_local_port_recv+0x17f/0x261
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8aedbef>] fc_local_port_recv_req+0x0/0x15b
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<c031fe2a>] _spin_unlock_irq+0x5/0x7
Jul 3 12:48:55 FCoE kernel: [<f8af2f0d>] fc_port_ingress+0x61/0x67
[openfctgt]
Jul 3 12:48:55 FCoE kernel: [<f8b04d54>]
fcoe_percpu_receive_thread+0x431/0x581 [fcoe]
Jul 3 12:48:55 FCoE kernel: [<c031fe2a>] _spin_unlock_irq+0x5/0x7
Jul 3 12:48:55 FCoE kernel: [<c031ec20>] __sched_text_start+0x398/0x405
Jul 3 12:48:55 FCoE kernel: [<f8b04923>]
fcoe_percpu_receive_thread+0x0/0x581 [fcoe]
Jul 3 12:48:55 FCoE kernel: [<c013713b>] kthread+0x34/0x55
Jul 3 12:48:55 FCoE kernel: [<c0137107>] kthread+0x0/0x55
Jul 3 12:48:55 FCoE kernel: [<c01059e7>] kernel_thread_helper+0x7/0x10
Jul 3 12:48:55 FCoE kernel: =======================
Jul 3 12:48:55 FCoE kernel: Code: <0f> 0b eb fe 8b 57 38 8d 47 38 8b 0a 39
c2 74 11 3b 72 10 74 04 89
Jul 3 12:48:55 FCoE kernel: EIP: [<f8ae4027>]
openfct_session_create+0x18/0xd1 [openfctgt] SS:ESP 0068:d1835e24
[Solution]
1. Perform FCoE Target setup first and then FCoE Initiator setup. i.e.,
Execute "fcconf ethx enable" command in FCoE Target, before executing "echo
"ethX" > /sys/module/fcoe/parameters/create" command in FCoE Initiator.
#6. [Issue]
------------
Execution of "fcconf eth0 enable" command fails with following error.
[r...@fcoe ~]# fcconf eth0 enable
fcconf: device file /dev/openfc does not exist- check udev and rules
[r...@fcoe ~]#
[Solution]
The "/dev/openfc does not exist" error message can be ignored. There is no
impact for FCoE target setup due to this error message.
_____________________________________________________________________
This e-mail message may contain proprietary, confidential or legally privileged
information for the sole use of the person or entity to whom this message was
originally addressed. Any review, e-transmission dissemination or other use of
or taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have received
this e-mail in error kindly delete this e-mail from your records. If it appears
that this mail has been forwarded to you without proper authority, please
notify us immediately at netad...@patni.com and delete this mail.
_____________________________________________________________________
_______________________________________________
devel mailing list
devel@open-fcoe.org
http://www.open-fcoe.org/mailman/listinfo/devel
