On Wed, 2010-09-01 at 15:49 -0700, Robert Love wrote: > On Tue, 2010-08-31 at 23:14 -0700, Bhanu Gollapudi wrote: > > On Tue, 2010-08-31 at 18:26 -0700, Robert Love wrote: > > > On Wed, 2010-08-25 at 01:31 +0000, Bhanu Gollapudi wrote: > > > > On systems with with higher nr_cpu_ids, per cpu exchange pool will > > > > have very limited exchange resources. This can cause certain > > > > operations such as discovery to fail even with finite amount > > > > of retires. This problem is handled by dividing the entire > > > > > > Hi Bhanu, > > > > > > Can you tell me a bit more about your scenario and how many CPUs > > > you're dealing with? Is it an offload EM or non-offload EM that's > > > running out of resources? > > > > Hi Robert, > > > > I described the scenario in one of the earlier emails > > http://www.mail-archive.com/[email protected]/msg07738.html > > > Ah. Thanks for the reference. > > I think this is a locking issue and not a resource problem. The lack of > resources definitely triggers the problem, but I think the system hangs > due to a deadlock. I don't think that discovery would should ever > completely fail. The discovery engine should retry 3 times and then the > lport state machine should restart causing discovery to restart. > Eventually discovery should succeed once the exchanges become available. > > I think that you pointed out the root cause of the hang in your initial > posting. I think that the disc_work context is trying to > cancel_delayed_work_sync() on itself. > > As the result of a exchange allocation failure fc_disc_error() is called > and a disc_work is scheduled with a delay. When then work is executed > fc_disc_timeout() is called and we're in the disc_work delayed work > context. This again fails to allocate an exchange for GPN_FT which calls > fc_disc_error() again. This time the retry_count has been exceeded and > disc_done() is called. This in turn calls fc_lport_disc_callback(FAILED) > which then calls fc_lport_reset(), which calls fc_lport_reset_locked(). > fc_lport_reset_locked() calls fc_disc_done() which calls > cancel_delayed_work_sync(&disc->disc_work). As you suspected this is > blocking as it tries to cancel the disc_work, which is the context that > we're in.
Absolutely, this is the real root cause, and it needs to be fixed. However running out of resources can lead into this hang quickly, and even if we fix it discovery takes a long time to complete after many retries, as we use only a subset of xid resources. Also, I was not very happy to see the xid resources shrink as the CPUs in the system goes higher, and hence submitted this change. Is there any downside of having the common pool? > > It is not an offload EM, and nr_cpu_ids in that system is 32, which > > leaves only 128 xids per cpu pool. > > > > > > > > The (non-offload) EM starts with 4k XIDs. It must be an extreme case if > > > it is running out of per-CPU XIDs. Especially if you're failing > > > discovery, when there shouldn't be that many exchanges in use. > > > > Please note that this is an extreme case, where there were 255 NPIV > > ports and the issue was triggered when a switch port > > shutdown/no-shutdown is performed. (There is another open issue in > > disc_stop which tries to cancel itself, described in the email) > > > > > > > > > exchange pool into two, one half to be used for cpu pool, and the > > > > other half to be used as common pool. If the exchange resource > > > > is not available in per cpu pool, the exchange resouce will > > > > be obtained from the common pool. > > > > > I think your problem is as described above, and fixing that locking > issue will resolve your failure. > > Separately, I'd be fine with some kind of resource scaling scheme. For > example if we allocate a lot of NPIV ports or have a lot of CPUs we grow > the XID pool. > > Thanks, //Rob > > _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
