On 7/23/10 1:23 AM, Bhanu Gollapudi wrote:
> I observed a system hang when the discovery fails due to lack of xid
> resources. This issue can happen when number of NPIV ports created are
> greater than the xids allocated per pool -- for eg., creating 255 NPIV
> ports on a system with nr_cpu_ids of 32, with each pool containing 128
> xids -- and then generating a link event - shutdown/no shutdown on the
> switch port causes the hang. Attached is the stack trace of the threads
> in blocked state.
>
>> From the stack trace,
>
> 1. disc_work that was scheduled as part of the 3rd retry attempt calls
> fc_disc_done with DISC_EV_FAILED.
> 2. disc_callback is called to complete the discovery with a failure.
> 3. holding lp_mutex fc_lport_reset_locked calls disc_stop
> 4. disc_stop tries to cancel the disc_work from the same work by calling
> cancel_delayed_work_sync(). (is it trying to cancel itself?)
>
> Please let me know your thoughts on this.
>
> Thanks,
> Bhanu
Good analysis.
The stack trace cancel problem could be fixed by noting that the discovery
is already stopped and disc_stop could return without doing anything. Some
new state may be needed, I haven't looked in detail.
The exchange problem could be handled by going to other pools when one is
empty. We could have a common pool to handle this case, or we could go
through all the other pools until we find an exchange, getting the proper
locks, of course. We might also want to use a larger exchange space when
there are more CPUs. After all, more CPUs usually means more memory as well.
Two-socket Westmere and hyperthreading gives 24 CPUs! This will become
more usual.
The LLD should tell libfc the maximum range of XIDs libfc is allowed
(taking into account offloaded exchanges), and libfc could either decide
at initialization time based on the number of CPUs how much of the range
to use, or grow the pool size as needed up to that maximum.
Also, maybe we shouldn't give up on discovery based on exchange exhaustion,
or at least give it a lot of retries. Wait a longer time in between retries,
perhaps.
Good work,
Joe
_______________________________________________
devel mailing list
[email protected]
http://www.open-fcoe.org/mailman/listinfo/devel
