[
https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157232#comment-13157232
]
Ralph Castain commented on MAPREDUCE-2911:
------------------------------------------
Ah - my bad. I didn't realize I was looking at the comments in reverse order :-)
After reading the comments in the correct order, I now better understand the
thread and see that Milind is following what I had suggested. As to the
discussion of secure communications, this is a continuing issue in the MPI
community. The problem is that securing at the message level creates
considerable overhead and severely impacts MPI performance.
What the community has chosen to do is secure at the user level, and then check
socket connections to ensure we are talking to someone from within our own
application. Thus, we launch based on ssh-like authentication requirements.
During MPI_Init, we wireup socket connections. As each connection is made, we
exchange an initial "ident" message that checks to ensure that the process on
the other end is a member of our application. If it isn't, we drop the
connection.
If you want to add further security during the socket formation phase, nobody
will object - though we might put it on a configuration basis so others aren't
impacted as it will slow down launch times on very large clusters.
HTH
Ralph
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Milind Bhandarkar
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI
> (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the
> past, running MPI application on a Hadoop cluster was achieved using Hadoop
> Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was
> kludgy. After the resource-manager separation from JobTracker in Hadoop, we
> have all the tools needed to make MPI a first-class citizen on a Hadoop
> cluster. I am currently working on the patch to make MPI an
> application-master. Initial version of this patch will be available soon
> (hopefully before September 10.) This jira will track the development of
> Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
