On Fri, 2010-09-03 at 11:23 -0700, Robert Love wrote: > > > > Absolutely, this is the real root cause, and it needs to be fixed. > > However running out of resources can lead into this hang quickly, and > > even if we fix it discovery takes a long time to complete after many > > retries, as we use only a subset of xid resources. Also, I was not very > > happy to see the xid resources shrink as the CPUs in the system goes > > higher, and hence submitted this change. Is there any downside of having > > the common pool? > > > No, not that I can think of, at least not with the non-offloaded EM. I > just wanted to point out that making the common pool wouldn't > necessarily _fix_ your problem, only make it less likely. > > I think that the OEM would not want the common pool though. There are > tricks that can be done to bind irqs to CPUs such that I/O stays on the > same CPU. This is done by ensuring that certain XIDs are always mapped > to certain CPUs. By creating a common pool for the OEM that mapping > isn't consistent for the common XIDs.
OK. > > I want to bring up a slightly unrelated topic as a talking point. I > discovered yesterday that NPIV is broken. This happened when we added > destination MAC address checking to fcoe.ko. The problem is that we're > only checking the N_Port's MAC address and not walking the list of NPIV > VN_Ports to check their MACs. The result is that we fail the validation > of all NPIV VN_Ports, so what you see on the wire is a successful FDISC, > a PLOGI to the name server and an ACC in response. The initiator then > sends an ABTS for the PLOGI to the name server because fcoe.ko prevents > libfc from seeing the ACC response. > > Yesterday, we debated a few solutions to this problem and the winning > solution seems to be to make the lport's VN_Port list read-safe so that > we can walk that list to check MACs without grabbing a lock (Chris' > idea). Currently, we only pass the N_Port lport up to libfc, find the > exchange from its EM and then call the upper layer callback (i.e. > fc_rport_plogi_resp()). It isn't until we're in the callback that we > look for the lport. Yes, currently fc_vport_id_lookup() is not exported, and then it holds lp_mutex. > > I realize that this isn't your problem, but what it does do is have > fcoe.ko pass the correct lport up to libfc (fc_exch_recv()). This means > that we don't need to rely on the exchange lookup to find the "real" > lport, which would allow us to have an EM per-lport. This allows for > scaling as NPIV ports are added, but doesn't really help with CPU > scaling. Yes, CPU scaling is still an issue. May be the number of XIDs can be a multiple of nr_cpu_ids, but if we have a system with a large number of cpus, we can exceed the XID limit of 64k. Another problem I observed with high number of NPIV ports is that, when they all fall in the same zone, each NPIV port tries to login to the other one causing it to establish (n*n-1) login sessions, which eventually leads to hang scenario. I was wondering, should we avoid the logins to NPIV ports if they belong to the same physical port? Thanks, Bhanu _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
