(Updated RFC, per offline discussion)
WHAT: Merge a tmp branch for fault recovery development into the OMPI
trunk
WHY: Bring over work done by Josh and Ralph to extend OMPI's fault
recovery capabilities
WHERE: Impacts a number of ORTE files and a ORTE ErrMgr framework
TIMEOUT: Barring objections and/or further requests for delay, evening
of March 23
REFERENCE BRANCH: http://bitbucket.org/jjhursey/orte-errmgr/
======================================================================
BACKGROUND:
Josh and Ralph have been working on a private branch off of the trunk
on extended fault recovery procedures, mostly impacting ORTE. The new
code optionally allows ORTE to recover from failed nodes, moving
processes to other nodes in order to maintain operation. In addition,
the code provides better support for recovering from individual
process failures.
Not all of the work done on the private branch will be brought over in
this commit. Some of the MPI-specific code that allows recovery from
process failure on-the-fly will be committed separately at a later
date. This commit provides the foundation for ORTE stabilization that
can be built upon to provide OMPI layer stability in the future.
This commit significantly modifies the ORTE ErrMgr framework to
support those advanced recovery operations. The ErrMgr public
interface has been preserved since it is used in various places
throughout the codebase, and should continue to be used as normal. The
ErrMgr framework has been internally redesigned to better support
multiple strategies for responding to failures (represents a merge of
the old ErrMgr and the RecoS framework, into the ErrMgr 3.0 component
interface). The default (base) mode will continue to work exactly the
same as today, aborting the job when a failure occurs. However, if the
user elects to enable recovery then one or more ErrMgr components will
be activated to determine the recovery policy for the job.
We have created a public repo (reference branch, above) with the code
to be merged into the trunk (r22815). Please feel free to check it out
and test it.
NOTE: The new recovery capability is only active if the user elects to
use it by setting the MCA parameter errmgr_base_enable_recovery to '1'.
NOTE: More ErrMgr recovery components will be coming online in the
near future, currently this branch only includes the 'orcm' module for
ORTE process recovery (not MPI processes). If you want to experiment
with this feature, below are the MCA parameters that you will need to
get started.
#################################
plm=rsh
rmaps=resilient
routed=cm
errmgr_base_enable_recovery=1
#################################
Comments, suggestions, and corrections are welcome!
On Mar 10, 2010, at 2:22 PM, Josh Hursey wrote:
Wesley,
Thanks for catching that oversight. Below are the MCA parameters
that you should need at the moment:
#####################################
# Use the C/R Process Migration Recovery Supervisor
recos_base_enable=1
# Only use the 'rsh' launcher, other launchers will be supported later
plm=rsh
# The resilient mapper knows how to use RecoS and deal with
recovering procs
rmaps=resilient
# 'cm' component is the only one that can handle failures at the
moment
routed=cm
#####################################
Let me know if you have any troubles.
-- Josh
On Mar 10, 2010, at 10:36 AM, Wesley Bland wrote:
Josh,
You mentioned some MCA parameters that you would include in the
email, but I don't see those parameters anywhere. Could you please
put those in here to make testing easier for people.
Wesley
On Wed, Mar 10, 2010 at 1:26 PM, Josh Hursey <jjhursey@open-
mpi.org> wrote:
Yesterday evening George, Thomas and I discussed some of their
concerns about this RFC at the MPI Forum meeting. After the
discussion, we seemed to be in agreement that the RecoS framework
is a good idea and the concepts and fixes in this RFC should move
forward with a couple of notes:
- They wanted to test the branch a bit more over the next couple of
days. Some MCA parameters that you will need are at the bottom of
this message.
- Reiterate that this RFC only addresses ORTE stability, not OMPI
stability. The OMPI stability extension is a second step for the
line of work, and should/will fit in nicely with the RecoS
framework being proposed in this RFC. The OMPI layer stability will
require a significant amount of work, but the RecoS framework will
provide the ORTE layer stability that is required as a foundation
for OMPI layer stability in the future.
- The purpose of the ErrMgr becomes slightly unclear with the
addition of the RecoS framework, since both are focused on
responding to faults in the system (and RecoS, when enabled,
overrides most/all of the ErrMgr functionality). Should the RecoS
framework be merged with the ErrMgr framework to create a new
ErrMgr interface?
We are typing to decide if we should merge these frameworks, but at
this point we are interested in hearing how other developers feel
about merging the ErrMgr and RecoS frameworks, which would change
the ErrMgr API. Are there any developers out there that are
developing ErrMgr components, or are using any particular features
of the existing ErrMgr framework that they would like to see
preserved in the next revision. By default, the existing default
abort behavior of the ErrMgr framework will be preserved, so the
user will have to 'opt-in' to any fault recovery capabilities.
So we are continuing the discussion a bit more off-list, and will
return to the list with an updated RFC (and possibly a new branch)
soon (hopefully end of the week/early next week). I would like to
briefly discuss this RFC at the Open MPI teleconf next Tuesday.
-- Josh
On Feb 26, 2010, at 8:06 AM, Josh Hursey wrote:
Sounds good to me.
For those casually following this RFC let me summarize its current
state.
Josh and George (and anyone else that wishes to participate
attending the forum) will meet sometime at the next MPI Forum
meeting (March 8-10). I will post any relevant notes from this
meeting back to the list afterwards. So the RFC is on hold pending
the outcome of that meeting. For those developers interested in
this RFC that will not be able to attend, feel free to continue
using this thread for discussion.
Thanks,
Josh
On Feb 26, 2010, at 6:09 AM, George Bosilca wrote:
On Feb 26, 2010, at 01:50 , Josh Hursey wrote:
Any of those options are fine with me. I was thinking that if
you wanted to talk sooner, we might be able to help explain our
intentions with this framework a bit better. I figure that the
framework interface will change a bit as we all advance and
incorporate our various techniques into it. I think that the
current interface is a good first step, but there are certainly
many more steps to come.
I am fine delaying this code a bit, just not too long. Meeting
at the forum for a while might be a good option (we could
probably even arrange to call in others if you wanted).
Sounds good, let do this.
Thanks,
george.
Cheers,
Josh
On Feb 25, 2010, at 6:45 PM, Ralph Castain wrote:
If Josh is going to be at the forum, perhaps you folks could
chat there? Might as well take advantage of being colocated, if
possible.
Otherwise, I'm available pretty much any time. I can't
contribute much about the MPI recovery issues, but can
contribute to the RTE issues if that helps.
On Thu, Feb 25, 2010 at 7:39 PM, George Bosilca <bosi...@eecs.utk.edu
> wrote:
Josh,
Next week is a little bit too early as will need some time to
figure out how to integrate with this new framework, and at
what extent our code and requirements fit into. Then the week
after is the MPI Forum. How about on Thursday 11 March?
Thanks,
george.
On Feb 25, 2010, at 12:46 , Josh Hursey wrote:
Per my previous suggestion, would it be useful to chat on the
phone early next week about our various strategies?
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
