Thanks Josh and Jeff. I have read already "The Design and Implementation of Checkpoint/Restart Process Fault Tolerance for Open MPI" and i'm gonna read now "A Composable Runtime Recovery Policy Framework Supporting Resilient HPC Applications" and then i will take a look to the code of the components that you mention, and i will let you know how things are going.
Thanks a lot. Hugo Meyer 2011/1/6 Joshua Hursey <jjhur...@open-mpi.org> > So I can point you to some of the work that I did while at Indiana > University to support process migration in Open MPI in a coordinated manner. > This should introduce you to some of the internal pieces that fit together > to provide this support. > > The transparent C/R in Open MPI webpage from IU is a good place to start: > http://osl.iu.edu/research/ft/ompi-cr/index.php > > >From there you will find a link to a couple papers that should get you > started. In particular "A Composable Runtime Recovery Policy Framework > Supporting Resilient HPC Applications" discusses how the ORTE ErrMgr > framework was used (initially) to provide process migration and automatic > recovery. The actual code in the Open MPI trunk is slightly different. > Instead of using different components of the ErrMgr framework (i.e., autor, > crmig, stable) we just rolled it all into the existing components (i.e., > hnp, orted, app). But all the code can be found in those component > directories. > > If you want a more general overview of the C/R system in Open MPI, I would > start with the paper "The Design and Implementation of Checkpoint/Restart > Process Fault Tolerance for Open MPI" which provides a high level view of > the architecture (combined with the paper above you will have a fairly > complete picture of the design). The C/R infrastructure currently only > supports coordinated C/R, but was designed to be more extensible. So if you > are looking into uncoordinated C/R techniques you may find that many of the > C/R frameworks in Open MPI can be reused. > > That should get you started. Let us know if you have any further questions. > > -- Josh > > On Jan 6, 2011, at 3:19 PM, Hugo Meyer wrote: > > > Thanks for the reply and don't worry about the delay. > > > > Yeah, i supposse it wouln't be easy :(. > > But my final goal is what you are mentioning, is to stop one particular > process (previously checkpointed) and the migrate it to another place (node, > core, slot, etc.) and restart it there, but without making a coordinated > checkpoint. I just need to checkpoint processes in an uncoordinated way, and > move them. > > > > Where can i see something about process migration in the code? or > something that could guide me. > > > > Greetings. > > > > Hugo Meyer > > > > 2011/1/6 Jeff Squyres <jsquy...@cisco.com> > > Sorry for the delay; you wrote while many of us were on vacation and > we're just now starting to catch up on past mails... > > > > I'm not entirely sure what you're trying to do. It sounds like you're > trying to replace one process with another. That's quite complicated; there > will be a lot of changes required in the code base to do this. > > > > - you'll need to notify the ORTE subsystem of the process change > > - this notification will likely need to span multiple processes > > - all MPI processes will need to quiesce their communications, > disconnect, and reconnect > > - ...and probably other things > > > > That being said, you might be able to leverage some of the work that's > been done with checkpoint/restart/migration. It's not entirely the same > thing that you're doing, but it's at least similar (quiesce networks, > [pretend to] move a process from location A to location B, etc.). > > > > > > > > On Dec 28, 2010, at 7:03 AM, Hugo Meyer wrote: > > > > > Hello to all. > > > > > > I'm new in the forum, at least is the first time i write. > > > > > > I'm working with open mpi and I would do a little experiment, i will > try to pass one process by another process. > > > > > > For example, assuming that there are 2 processes that are communicating > say rank 1 and 2. And there is a process of rank 3, I would like the rank 3 > (it could be assumed that this node is marked down at the initial hostfile) > took the place of rank 2, and rank 1 still think that he is communicating > with rank 2 when in fact is communicating with the rank 3. > > > > > > I guess I'll have to modify tables as orte_job_map_t and orte_proc_t, > but I wanted to know if someone already has experience doing something > similar, and can guide me at least. > > > > > > The communication between processes, in principle, would be irrelevant, > so i will not need to use checkpoints / restarts for now. > > > > > > Greetings > > > > > > Hugo Meyer > > > _______________________________________________ > > > devel mailing list > > > de...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
