[
https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277916#comment-13277916
]
Milind Bhandarkar commented on MAPREDUCE-2911:
----------------------------------------------
I am excited to report that, thanks to great efforts by Ralph Castain and
Wangda Tan, Hamster (i.e. OpenMPI on Yarn) now works flawlessly, and is
scheduled to be merged to OpenMPI trunk soon. This effort was equivalent to
building a second floor on a mobile home while it was hurtling down the freeway
at 65 MPH :-) Thanks to both Ralph & Wangda.
According to Ralph:
"Lots of cleanup and documentation to do, and performance sucks per HPC
standards. But at least it works!"
To my knowledge, this is the first application framework implemented in C that
uses the multi-lingual protobuf APIs for Yarn. (For secure environments, a
small java-based shim is needed.)
Also, it is encouraging that no changes were needed in Yarn to make resource
allocation work for MPI. (MPI as a standard came along in 1994, 18 years before
Yarn was designed.)
Currently, using MPI-IO functionality in MPI requires a shared posix
file-system mounted on every node. However, this will change in future. For
some distributed file systems (*cough*), which offer posix interface, MPI-IO
works today.
Once it is decided whether BigTop can include Non-ASF packages, we plan to work
with BigTop community to integrate OpenMPI (new BSD-licensed) in the big data
stack.
I am closing this issue as fixed.
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI
> (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the
> past, running MPI application on a Hadoop cluster was achieved using Hadoop
> Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was
> kludgy. After the resource-manager separation from JobTracker in Hadoop, we
> have all the tools needed to make MPI a first-class citizen on a Hadoop
> cluster. I am currently working on the patch to make MPI an
> application-master. Initial version of this patch will be available soon
> (hopefully before September 10.) This jira will track the development of
> Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
