This is caused by a complex interaction of ib_mad, hcad_mod and pSeries firmware.
As you might already have noticed a eHCA doesn't show up as a "port" but as a switch in the fabric.
Reason for this is partition support and virtualisation in Infininband.
If you want to give each partition in a system a "own" IB adapter, it has to have its "own" LID(s) and therefore it's own GUIDs.
IB standard only allows one way currently how to accomplish this: You need a switch and multiple adapters behind.
So that's exactly how the eHCA shows up in the fabric. In our case system firmware handles the SMA traffic for that "switch" and for all "adapters" (running an SMA or SM on QP0 is currently not supported).
This brings up another problem: you definetly won't want to allocate LIDs for all "potentially possible" operating system partitions (not to confuse with IB partitioning), otherwise you could come close to the 48000 LIDs/subnet limit pretty quickly. So you need some kind of signal from the operating system to system firmware,
which in the eHCA case is the H_DEFINE_AQP1 triggered by ib_create_qp with IB_QPT_GSI parameter.
AFTER that call handshaking between system firmware and the SM will start, here's a new adapter active on a switch port... what's your guid? here's your LID, p_key, SM lid....
...and after all that it's possible to send and receive packets from the fabric.
The openib stack expects that a port is fully functional after this create_qp returns, and starts to do all sorts of modify QP and post send.
So the only choice we have there is to delay create_qp until the complete handshaking between system firmware and the SM has finished (until we see a IB_PORT_ACTIVE in hcad_mod). If we don't see that until EHCA_PORT_ACTIVE_TIMEOUT we have to return an error code to openib, otherwise we're seriously in trouble (tried that).
Shirley already pointed out on the mailinglist, that ib_mad and others have different recovery depending on the success of ib_create_qp(IB_QPT_GSI), especially ib_mad decides it's the best thing to kill the complete adapter if that call fails on a single port.
so that's the full explanation of ehca_nr_ports and hopefully answers your question....
| Troy Benjegerdes <[EMAIL PROTECTED]>
08.10.2005 04:03 |
|
On Fri, Oct 07, 2005 at 09:33:27AM -0700, Shirley Ma wrote:
> Hi, Troy,
>
> There is INSTALL file in the EHCA driver package.
> In OpenPower 720 port 1 is at the top, port 2 is at the bottom.
> In P570, port1 is at the bottom, port2 is at the top.
Okay, I guess I should read more carefully ;)
What is the issue with needing to use port 1? Can that be fixed in the
driver, or does that need a firmware update?
_______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
