[EMAIL PROTECTED] wrote on Thu, 21 Sep 2006 13:40 -0400:
> And the non-NULL will be a mx://... string, no?
No, this is the bmi_method_addr that comes back from your lookup
method. You cast it up into your own private address structure,
and do not have to parse the string again.
> As for listening on all boards, that is not supported by MX directly
> (a MX endpoint is specific to a NIC). If this is a requirement of
> PVFS (to be to use multiple Myricom NICs), then we have a couple of
> options. First, can PVFS open two BMI methods of the same type? If
> so, then we just have to ensure that all bmi_mx state is local to
> each process (no globals, etc.). If BMI cannot open two of the same
> type, then I will have to manage two (or more) MX endpoints in bmi_mx.
I'm not sure if that's possible. BMI does fully support
"multi-homed" networks on _different_ types (like IB or TCP), but
I don't think it can do what you're thinking, nor would you want it
to do that.
Consider BMI_test_unexpected. Would you want BMI to call that
separately for each board, or can you instead check each board
yourself? BMI currently expects the latter. Also there's some
features that would likely be global across all BMI_mx instances,
like a lookup table that would convert a peer BMI address into a
local board number. That would be a bit unclean.
> That then raises the issue of whether the NICs are in the same fabric
> or in disjoint fabrics. If the same, then I can send (and receive)
> using either one and we will need to consider some form of striping
> over the NICs to ensure that we maximize utilization of both. If they
> are disjoint, I will need to maintain separate peer lists for each
> and determine which NIC needs to send and receive for a given peer.
Fun.
> Using more than one board definitely complicates matters. ;-)
> Initially, I will support one board to get things going.
Good plan. Then figure out later if somebody wants to do a setup
like this. But keep it in mind as you think about the design
decisions.
> I am not worried about peer (i.e. server) state at initialize time.
> Sending to others is not an issue as long as the URI is passed as
> part of the send context. Does the client also get a listen_addr
> string or not? I would assume not. If so, I am then free to open any
> endpoint I want and then I would pass that info to the server before
> sending my first sendunexpected message (i.e. in my method's connect
> request message).
That's how you tell if you're a server: listen_addr == NULL => I am
a client. Sounds like this is the easy side of the communication.
> If the client does not get a listen_addr URI and if the machine has
> mutiple NICs, I will not know which one to open. I could have a
> #define that compiles in the board number but it would require that
> all machines use the same board. Any suggestions?
Don't open anything until somebody calls BMI_mx_sendunexpected().
You could probe to make sure a NIC exists, but don't commit to using
it until you need to do so. Or punt like on the server side. IB
assumes only a single NIC for now, too. Most people aren't crazy
enough to use more than one of these things in a machine anyway.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
