On 04/27/2017 06:02 AM, Neale Ferguson wrote:
Thanks Mark. One PITA with the new naming scheme is that it is still
different from LinTel as the WWN and LUN components are prefixed by 0x on
s390x but not so on LinTel.
I would be very interested in some example output of "ls -laF
/dev/disk/by-path/" of both x86 and s390x.
Currently, I cannot imagine how the "new" (for zfcp) naming scheme could
be any different on architectures, because zfcp now makes use of the
very same common code creating the strings for WWPN and LUN which has
always been there for x86 et al.
See also below for common code by-path formatting of LUNs <256 which is
indeed decimal rather than hex (on any architecture).
On 4/26/17, 11:06 PM, "Linux on 390 Port on behalf of Mark Post"
<LINUX-390@VM.MARIST.EDU on behalf of mp...@suse.com> wrote:
On 4/26/2017 at 09:43 PM, Neale Ferguson <ne...@sinenomine.net>
wrote:
What level of s390-utils changed the zFCP device naming from:
ccw-<device_bus_id>-zfcp-<wwpn>:<lun> or
ccw-<device_bus_id>-zfcp-<wwpn>:<lun>-part<n>
to
/dev/disk/by-path/ccw-<device_bus_id>-fc-<wwpn>-lun-<lun>-part<n>
We had a bug reported against this for SLES12 SP2. It was against
systemd/udev. The version we have is udev-228.
SLES12 SP2 got the systemd change via upstream and s390-tools ships a
compat udev rule /lib/udev/rules.d/59-zfcp-compat.rules, in order to
_additionally_ create the old zfcp-specific by-path naming scheme.
# lsscsi
[0:0:1:635] disk IBM 2145 0000 /dev/sda
[1:0:0:1087324290] disk IBM 2107900 1400 /dev/sdc
# lsscsi -x
[0:0:1:0x027b_0000] disk IBM 2145 0000
/dev/sda
[1:0:0:0x4082] disk IBM 2107900 1400
/dev/sdc
^^^^ not unique because T10 SAM does not define 2nd level here
# lsscsi -xx
[0:0:1:0x027b000000000000] disk IBM 2145 0000
/dev/sda
[1:0:0:0x408240cf00000000] disk IBM 2107900 1400
/dev/sdc
# l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 Apr 25 19:02
ccw-0.0.50c0-fc-0x500507680b2281fb-lun-0x027b000000000000 -> ../../sda
lrwxrwxrwx 1 root root 9 Apr 25 19:02
ccw-0.0.50c0-zfcp-0x500507680b2281fb:0x027b000000000000 -> ../../sda
lrwxrwxrwx 1 root root 9 Apr 25 19:02 scsi-0:0:1:635 -> ../../sda
lrwxrwxrwx 1 root root 9 Apr 27 12:06
ccw-0.0.1900-fc-0x50050763070845e3-lun-0x408240cf00000000 -> ../../sdc
lrwxrwxrwx 1 root root 9 Apr 27 12:06
ccw-0.0.1900-zfcp-0x50050763070845e3:0x408240cf00000000 -> ../../sdc
lrwxrwxrwx 1 root root 9 Apr 27 12:06 scsi-0:0:0:1087324290 -> ../../sdc
Ubuntu 16.04 got the systemd change via upstream but does not ship a
compat rules because as the first release on s390x there is no regression.
# lsscsi
[0:0:0:1087389826]disk IBM 2107900 1400 /dev/sda
[2:0:1:607] disk IBM 2145 0000 /dev/sdj
# lsscsi -x
[0:0:0:0x4082] disk IBM 2107900 1400
/dev/sda
^^^^ not unique because T10 SAM does not define 2nd level here
[2:0:1:0x025f_0000] disk IBM 2145 0000
/dev/sdj
# lsscsi -xx
[0:0:0:0x408240d000000000] disk IBM 2107900 1400
/dev/sda
[2:0:1:0x025f000000000000] disk IBM 2145 0000
/dev/sdj
# l /dev/disk/by-path/
lrwxrwxrwx 1 root root 9 Apr 25 16:15
ccw-0.0.1900-fc-0x50050763070845e3-lun-0x408240d000000000 -> ../../sda
lrwxrwxrwx 1 root root 9 Apr 27 12:12
ccw-0.0.50c0-fc-0x500507680b2281fb-lun-0x025f000000000000 -> ../../sdj
(I assume it is a udev rule that does this - if so which one?)
I can't find any specific rule that does this. I suspect the code that
did this is in src/udev/udev-builtin-path_id.c
Yes, upstream commit:
https://github.com/systemd/systemd/commit/e7eb5a8d88367a755944fdda3023a308e5272953.
Also, does the FCP standard define the size of lun?
It's T10 SAM, the SCSI Architecture Model, which defines size (64 bit)
and formats of LUN. [http://www.t10.org/drafts.htm#SCSI3_ARCH]
In the kernel header
lun is u64
The Linux kernel always handles LUN as opaque u64 without trying to
parse or interpret any of the different LUN formats.
However, since T10 SAM defines the u64 LUN as composed of up to 4
(hierarchical) levels, the Linux kernel uses a special bijective mapping
to represent the 64 bit binary LUN as a "handy" decimal "SCSI LUN" to
user space. The user can see this as the fourth part in the "HCTL" name
of a SCSI device consisting of <scsi host>:<scsi bus>:<scsi
target>:<scsi lun> all of which are decimal numbers. This is the Linux
SCSI stack midlayer naming scheme which exists for any SCSI transport
includig but not limited to FCP.
With FCP, the FCP device maps to a scsi host (unpredictably numbered
from zero on dynamic host allocation),
with zfcp, the scsi bus is always 0 (some parallel SCSI host bus
adapters having >= 2 ports accessed over the same hardware device chip
use this),
with FCP, the target port (fc_rport) maps to a scsi target
(unpredictably numbered from zero on dynamic port discovery and allocation).
The LUN levels in a hex representation are 0x1111222233334444. Since the
use starts at level 1 and expands to higher levels depending on the LUN
format, the smallest LUNs in hex representation starting with LUN 0 and
incrementing by one are:
0x0000000000000000
0x0001000000000000
0x0002000000000000
0x0003000000000000
...
In order to be somewhat compatible to the old parallel SCSI transport
(T10 SPI) [http://www.t10.org/drafts.htm#SCSI3_PAR,
http://www.t10.org/scsi-3.htm], Linux uses half-word (u16) shuffling to
have (single level) peripheral device addressing LUNs appear with the
same small decimal numbers starting at zero as SPI. While it might look
like endianess conversion, it is different. So above numbers appear as
Linux scsi lun:
0
1
2
3
...
By-path names for FCP in the common code naming scheme, happen to format
LUN as decimal if the LUN value is <256, else as full 16 hex digit with
leading '0x'.
https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-path_id.c#L65
function format_lun_number()
NB: Since a while the Linux kernel does support full 64 bit LUNs, so the
comment in systemd regarding (upper) 32 bit LUNs is kind of outdated and
it seems systemd by-path could not properly format such LUN if it made
use of LUN levels 3 or 4.
# lsscsi
[0:0:1:1] disk IBM 2145 0000 /dev/sdb
# lsscsi -x
[0:0:1:0x0001] disk IBM 2145 0000
/dev/sdb
# lsscsi -xx
[0:0:1:0x0001000000000000] disk IBM 2145 0000
/dev/sdb
# l /dev/disk/by-path/total 0
lrwxrwxrwx 1 root root 9 Apr 27 11:56
ccw-0.0.50c0-fc-0x500507680b2281fb-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root 9 Apr 27 11:56 scsi-0:0:1:1 -> ../../sdb
while the compat rule shipped with SLES12 SP2 would additionally create
the old zfcp-specific always hex name:
lrwxrwxrwx 1 root root 9 Apr 27 11:56
ccw-0.0.50c0-zfcp-0x500507680b2281fb:0x0001000000000000 -> ../../sdb
but in non-z architectures lun tends to be a number between 0
and 255.
This seems an urban legend I would like to get rid of.
The format of a LUN has nothing to do with the initiator (host) or its
architecture.
It is solely the storage target which decides on a LUN format and
exports LUNs in this format (see also REPORT LUNS SCSI command [T10 SPC]
which is used for automatic LUN scanning on any initiator architecture).
[http://www.t10.org/drafts.htm#SPC_Family]
See also Slide 10 "Logical Unit Numbers (LUNs)" in
http://www-05.ibm.com/de/events/linux-on-z/pdf/day2/4_Steffen_Maier_zfcp-best-practices-2015.pdf.
The only indirect relation to the initiator is with some storage target
types, such as IBM DS8000: To export LUNs to an initiator, one defines
"host mappings" based on initiator WWPNs (typically the NPIV WWPNs of
FCP devices on s390x). A host mapping or volume group has some property
regarding the initiator "type", in order to chose some specific settings
regarding target behavior to work optimally with the initiator type.
The host type "Linux on z" happens to also select the LUN mapping scheme
"SCSIMASK" as property resulting in 0x40xx40yy00000000, where xx is the
LSS and yy the storage internal volume ID within this LSS. This is
pseudo flat space addressing but with 2nd level, which can address 65536
LUNs.
Other host types, including Linux on x86, happen to select the LUN
mapping scheme "SCSIMAP256" resulting in 0x00zz000000000000, where zz is
a freely selectable 256 bit number resulting in an independent mapping
of internal volume ID xxyy to external LUN zz. This is T10 SAM
peripheral device addressing LUN format, which can only address 256 LUNs.
While not qualified, one could export LUNs with SCSMAP256 to Linux on z,
but I would not recommend it.
Other storages, such as IBM Storwize, do not use this initiator type
distinction, and so even Linux on z typically sees LUN formats of type
peripheral device addressing (and for LUNs > 256 FlashSystems switches
to a (single level) flat space addressing scheme of 0x4zzz00000000 which
can address 16384 LUNs).
Storages sometimes also use the bus identifier field of (single level)
peripheral device addressing LUNs to address more than the strictly
speaking possible 256 LUNs [e.g. 0x025f000000000000 above].
--
Mit freundlichen Grüßen / Kind regards
Steffen Maier
Linux on z Systems Development
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
