On Tue, 2010-08-31 at 23:14 -0700, Bhanu Gollapudi wrote: > On Tue, 2010-08-31 at 18:26 -0700, Robert Love wrote: > > On Wed, 2010-08-25 at 01:31 +0000, Bhanu Gollapudi wrote: > > > On systems with with higher nr_cpu_ids, per cpu exchange pool will > > > have very limited exchange resources. This can cause certain > > > operations such as discovery to fail even with finite amount > > > of retires. This problem is handled by dividing the entire > > > > Hi Bhanu, > > > > Can you tell me a bit more about your scenario and how many CPUs > > you're dealing with? Is it an offload EM or non-offload EM that's > > running out of resources? > > Hi Robert, > > I described the scenario in one of the earlier emails > http://www.mail-archive.com/[email protected]/msg07738.html > Ah. Thanks for the reference.
I think this is a locking issue and not a resource problem. The lack of resources definitely triggers the problem, but I think the system hangs due to a deadlock. I don't think that discovery would should ever completely fail. The discovery engine should retry 3 times and then the lport state machine should restart causing discovery to restart. Eventually discovery should succeed once the exchanges become available. I think that you pointed out the root cause of the hang in your initial posting. I think that the disc_work context is trying to cancel_delayed_work_sync() on itself. As the result of a exchange allocation failure fc_disc_error() is called and a disc_work is scheduled with a delay. When then work is executed fc_disc_timeout() is called and we're in the disc_work delayed work context. This again fails to allocate an exchange for GPN_FT which calls fc_disc_error() again. This time the retry_count has been exceeded and disc_done() is called. This in turn calls fc_lport_disc_callback(FAILED) which then calls fc_lport_reset(), which calls fc_lport_reset_locked(). fc_lport_reset_locked() calls fc_disc_done() which calls cancel_delayed_work_sync(&disc->disc_work). As you suspected this is blocking as it tries to cancel the disc_work, which is the context that we're in. > It is not an offload EM, and nr_cpu_ids in that system is 32, which > leaves only 128 xids per cpu pool. > > > > > The (non-offload) EM starts with 4k XIDs. It must be an extreme case if > > it is running out of per-CPU XIDs. Especially if you're failing > > discovery, when there shouldn't be that many exchanges in use. > > Please note that this is an extreme case, where there were 255 NPIV > ports and the issue was triggered when a switch port > shutdown/no-shutdown is performed. (There is another open issue in > disc_stop which tries to cancel itself, described in the email) > > > > > > exchange pool into two, one half to be used for cpu pool, and the > > > other half to be used as common pool. If the exchange resource > > > is not available in per cpu pool, the exchange resouce will > > > be obtained from the common pool. > > I think your problem is as described above, and fixing that locking issue will resolve your failure. Separately, I'd be fine with some kind of resource scaling scheme. For example if we allocate a lot of NPIV ports or have a lot of CPUs we grow the XID pool. Thanks, //Rob _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
