Re: [OMPI devel] Preparations for moving the btl's

[Ralph Castain]

On Dec 4, 2008, at 7:05 AM, Jeff Squyres wrote:
After reflecting on this a bit, there's two more things I should  
have mentioned:

1. I think that moving the BTL's out into their own layer (or  
whatever) should be a separate effort than re-introducing the RSL  
(or something like it).  To me, they're two different things and  
should be addressed separately.
2. With both the BTLs and some incarnation of RSL coming into the  
code base, we need to decide exactly what our policies will be on  
who can drive interface changes and what our responsibilities will  
be to external code bases that use the BTL and/or RSL interfaces.

FYI: Re-introducing the some form of the RSL is already on either  
the December or January ORTE meeting agenda (I don't remember which  
offhand).

We haven't scheduled the post-Dec meeting yet - perhaps we should? If  
you look at the wiki, there is no way that topic is going to get  
covered in Dec - too many other higher-priority topics, too little time.

Ralph




On Dec 4, 2008, at 8:13 AM, Jeff Squyres wrote:

I think you got it right.  And I think we're pretty good in terms  
of BTL usage of ORTE and OPAL (to include the new "notifier"  
service that Ralph put in recently -- what the FTB will likely  
eventually use, I think...?); those interfaces and abstraction  
barriers are technologically enforced.  If you break the  
abstractions, the linker will swiftly and unmercifully punish you.   
(this was exactly [one of] the rationale that we used for splitting  
the code base into OPAL, ORTE, and OMPI several years ago)

Greg has already noted on the wiki a few constants used in the  
BTL's that have an OMPI_ prefix that aren't really OMPI values  
(e.g., OMPI_ENABLE_HETEROGENEOUS_SUPPORT).  These come from  
configure (i.e., opal/include/opal_config.h) and were not renamed  
back when we split the code base into OPAL, ORTE, and OMPI.  I  
don't think we had a strong reason for not renaming them -- most  
could probably be renamed to OPAL_* -- we just didn't do it then.   
Perhaps they can be changed during the BTL extraction process (I  
noted this on the wiki).



On Dec 3, 2008, at 9:43 PM, Richard Graham wrote:

BTW,
I was guessing FTB is Fault Tolerant Backbone, but if not, can  
someone tell me what it is ?  If it is not the later, what I just  
wrote about it makes no sense.

Rich


On 12/3/08 9:34 PM, "Richard Graham" <rlgra...@ornl.gov> wrote:

The goal is to use the btl’s outside of the context of MPI, which  
was what was in mind from the day the ompi work started over five  
years ago, but with no other use at the time, things grew up  
intermingled – no surprise at all.  What we are attempting to do  
is to untangle the existing dependencies, and make a much cleaner  
distinction between how/what data is passed between layers.

I expect this will involve some sort of well defined interface  
between the btl’s and orte, and I don’t know if this will also  
require something like this between the btl’s and the pml – I  
think that interface is rigidly enforced, but am not sure.

I expect that explicit calls to FTB in the btl layer would have  
to be componentized, especially in the context of what is  
developing in the FT working group of the MPI Forum.  Not that  
FTB is bad in any way, just that it is one of many monitors.

We will need to talk about this on a case by case basis, and  
decide how to proceed.  If anyone wants to help, please do.

Rich


On 12/3/08 3:02 PM, "Ralph Castain" <r...@lanl.gov> wrote:

I managed to execute the modex-less changes pretty much without
introducing additional ORTE dependencies into the BTL's, though  
there
may be some additions as we look a the other BTLs that I didn't
address. So hopefully that won't contribute too much to the  
issue here.

At the moment, I don't think it matters where notifier sits - it  
might
be able to move to OPAL. Only catch will be if some notifier  
component
requires communications. I'm thinking of FTB, for example, and  
our own
local monitoring program that may require TCP messaging. We don't
currently have anything in OPAL that would support an OPAL level
messaging system, though perhaps that could be resolved.

We also have dependencies where the BTL's will call orte_ess to  
find
out what node another proc is on, the node local rank of that  
proc,
etc. Those dependencies are likely to grow after the Dec meeting  
(see
wiki for that agenda item), and definitely cannot be moved to  
OPAL.

However, note that Rich stated the BTL's were -not- moving to  
OPAL.
This begs the question: where -are- they going? Into their own  
layer?
Will that layer be somewhere in-between OMPI and ORTE (in which  
case,
the ORTE dependencies are moot)?

I note that the wiki page doesn't address any of these questions,
which is understandable if things are just getting underway. But  
it
does sound like this is going to take some thought to ensure we  
don't
paint ourselves into a corner.

Ralph


On Dec 3, 2008, at 12:10 PM, Jeff Squyres wrote:

> FWIW, I see lots of notifier calls being added to the BTLs (and
> elsewhere throughout the OMPI code base) over time...
>
> On Dec 3, 2008, at 2:07 PM, Tim Mattox wrote:
>
>> The BTLs might have added calls to the notifier framework in  
their
>> error paths.
>> The notifier framework is currently in the ORTE layer... not  
sure
>> if we could
>> move it down to OPAL.  Ralph, any thoughts on that?
>>
>> On Wed, Dec 3, 2008 at 11:56 AM, Richard Graham <rlgra...@ornl.gov 
>
>> wrote:
>>> George told me about what he is doing, so no changes would be
>>> committed
>>> until George has his changes in.
>>>
>>> Are there other changes to the btl's that we should be aware  
of ?
>>>
>>> Rich
>>>
>>>
>>> On 12/3/08 11:47 AM, "George Bosilca" <bosi...@eecs.utk.edu>  
wrote:
>>>
>>>> Terry,
>>>>
>>>> I'm involved [at some degree] in both efforts and I can  
confirm
>>>> these
>>>> two efforts will not affect each other in any bad way.
>>>>
>>>>  george.
>>>>
>>>> On Dec 3, 2008, at 11:42 , Terry Dontje wrote:
>>>>
>>>>> I don't have any *strong* objections. However, I know that  
Eugene
>>>>> and George B have been working on some Fastpath code changes
>>>>> that we
>>>>> should make sure neither project obliterates the other.
>>>>>
>>>>> --td
>>>>>
>>>>> Richard Graham wrote:
>>>>>> Now that 1.3 will be released, we would like to go ahead  
with the
>>>>>> plan to move the btl’s out of the MPI layer. Greg Koenig  
who is
>>>>>> doing most of the work has started a wiki page with  
details on
>>>>>> the
>>>>>> plans. Right now details are sketchy, as Greg is digging  
through
>>>>>> the code, and has only hand written notes on data  
structures that
>>>>>> need to be moved, include files that are not needed, etc.  
The
>>>>>> page
>>>>>> is at:
>>>>>> _https://svn.open-mpi.org/trac/ompi/wiki/BTLExtraction_
>>>>>>
>>>>>> The first three steps basically only involve code motion,  
moving
>>>>>> items such as ompi_list, and renaming them, moving where  
the code
>>>>>> is actually located in the repository, and the like. For  
these we
>>>>>> do not plan to put out a formal RFC, but comments are very
>>>>>> welcome,
>>>>>> and any hands that are willing to help with this are even  
more
>>>>>> welcome.
>>>>>>
>>>>>> The last phase where the btl’s are made dependent on  
OPAL, and
>>>>>> supporting libraries such as mpools I expect will be  
disruptive,
>>>>>> and will definitely require an RFC, and will also be a  
longer
>>>>>> process.
>>>>>>
>>>>>> Please send comments,
>>>>>> Rich
>>>>>>  
------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> de...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>
>>
>>
>> --
>> Tim Mattox, Ph.D. - http://homepage.mac.com/tmattox/
>> tmat...@gmail.com || timat...@open-mpi.org
>>   I'm a bright... http://www.the-brights.net/
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> Cisco Systems
>
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Jeff Squyres
Cisco Systems


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Jeff Squyres
Cisco Systems


_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel