To put it another way - When running a FCP chpid in Non-NPIV mode - the demands made on the channel by the CPC and made on the SAN switch by the channel are very low. You can totally run 256 subchannels in a single LPAR, or like 1000 something in total across multiple LPARs using a single FCP port to the SAN, since as far as the SAN is concerned there is only one Server on that port logging in and running I/O.
This is a pain to manage from the SAN side however since you can't control which guest or LPAR has access to which storage device since it is all one big server due to the lack of NPIV. There are lots of opportunities to fat finger something and blow away someone else's stuff here. Once you tun NPIV on for that FCP channel though, each active subchannel with an NPIV WWPN consumes additional memory and CPU resources within the FCP channel adapter, and also consumes additional memory and CPU resources in the attached SAN switch. This is most apparent when doing a lot of IPLs or a switch reboot as Alan mentioned. In one of my test environments, we are a little ( a lot! ) beyond the edge of the 32 NPIV subchannel limit ( which is documented in the system Z IODF books from what I remember ) and when we do a mass IPL of all guests - some ( 10% ish ) of the guests come up with no SCSI volumes on their FCP device. The simple fix for us is to go back and reIPL the guests missing SCSI volumes one at a time and wait 5 minutes between each IPL, but we are a test lab and can tolerate that kind of thing. So - the moral of the story: when running on a NPIV-enabled FCP channel limit the total number of users ( both Linux and VM EDEV ) of that Channel to 32 ( or 64 on a z13 which as faster bigger FCP adapters ) unless you like experiencing randomish SAN fabric timeouts during mass IPLs or after applying switch Firmware - or anything that drives a lot of SAN FLOGI activity from each NPIV user. On Thu, Apr 30, 2015 at 3:25 PM, Alan Altmark <[email protected]> wrote: > On Thursday, 04/30/2015 at 12:43 EDT, Mike Walter <[email protected]> > wrote: > > Regarding your previous warning: just don't put more than 32 guests (64 > for > > the z13) on each FCP chpid. Doing so can result in failure to fully > recover an > > FCP chpid after an error. > > > > Where is that recommendation documented in more detail? Curious minds > want to > > know. > > > > The subject thread was about EDEVs, but I'd like to understand if there > is a > > larger implication. Is the 32 guests per (non-z13) FCP chpid specific > to > > EDEVs, or would it include FCP rdev SAN access, too? > > Yes, it include FCP rdevs, too, but I should have mentioned that the limit > applies to NPIV configurations only. (What VM shop would be using > non-NPIV?) > > From the z13 announcement: > > "With the introduction of the FICON Express16S in an IBM z13 operating > using the FCP protocol, several recommended and allowable operating > characteristic values have increased which will enable additional workload > consolidation. Specifically, the recommended maximum number of NPIV hosts > defined to any single physical FCP channel has increased from 32 to 64, > the allowable maximum number of remote N_Ports a single physical channel > can communicate with has increased from 512 to 1024, and the maximum > number of LUNs addressable by a single physical channel has increased from > 4096 to 8192. In support of these increases, the FCP channels have also > been designed to now support 1528 concurrent I/O operations, an increase > from the prior generation FCP channel limit of 764. " > > I believe that the original recommendation was in a Redbook; it's not in > the Machine Limits section of the IOCP book, but it is mentioned in the > FCP discussion in the front of the IOCP book (~p.35). It might be > tempting to go to non-NPIV to get around it, but keep in mind that two > subchannels (RDEVs) on the same non-NPIV chpid will be unable to open the > same SCSI LUN via the same target WWPN. > > > E.g. our current environment includes: > > > > * Each guest has two 3390-27 ECKD DASD to house the Linux OS and > core > > software common to all servers (not my disk allocation, I get no voice > in that). > > > > * Each guest has two dedicated FCP rdevs (not EDEVs) on > different > > chpids (for redundancy) providing access to SAN LUNs which house their > Linux > > apps and databases. > > > > * Each LPAR has 64 (or 60) FCP rdevs defined on each of the 9 > FCP > > chpids on each LPAR (actual max = 575 FCP rdevs per LPAR). Limiting > each FCP > > chpid to 32 guests would currently limit us to approximately 288 guests > per > > LPAR. We're not close to 288 guests yet, but typically once "they" have > seen a > > Linux on z Systems guest running, "they" want more... as fast as > possible. > > > > We are running rather low on ECKD DASD to house the OS and core software > for > > new servers, so we have begun evaluating use of EDEVs in place of ECKD > disk. > > If we added two EDEVs (one primary, and one on a separate chpid), how > would the > > FCP rdevs and the FCP EDEVs be counted? Is it truly only EDEVs that are > > counted towards the (non-z13) 32 guest limit, or do we count the FCP > rdevs for > > LUN access in that count, too? > > Eh? Low on dasd? Operators are standing by...... > > When a SAN switch goes down, is rebooted or whatever, all of the > connected guests will get I/O errors and switch to the other path on > another switch. Fine. Now the switch finishes rebooting or is powered on > or replaced. "Yaaaay!" say the guests. "The new adapter is here! The > new adapter is here!" And they all activate their connections to the > remote LUNs. allatthesametime. The switches don't always appreciate > the flood. > > Alan Altmark > > Senior Managing z/VM and Linux Consultant > Lab Services System z Delivery Practice > IBM Systems & Technology Group > ibm.com/systems/services/labservices > office: 607.429.3323 > mobile; 607.321.7556 > [email protected] > IBM Endicott > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- Jay Brenneman ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
