[
https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251891#comment-13251891
]
Arun C Murthy commented on MAPREDUCE-2911:
------------------------------------------
Ralph, thanks again for the discussion.
If I wasn't clear before, I'd like to re-emphasize that I look at Hamster as a
way to bring MPI apps to the Hadoop world in a first-class manner (as opposed
to the hacks employed so far), not necessarily as a way to bring YARN to the
MPI world to which I'm agnostic (too).
I fully respect that the HPC world has it's own conventions and expectations
which isn't really what YARN is geared to solve.
As a result, with some engineering effort I'm confident Hamster could be a
really useful addition to the Hadoop toolkit even with some of the current
issues you pointed out.
----
bq. MPI jobs typically do run for long times, though surveys report that
roughly 70% of them run for one hour or less (but more than a few minutes).
Like I said, I'm sure it matters in the HPC world, but I'm pretty sure it's
much less of an issue in the Hadoop world even as we work to improve container
launch etc. As a result, I'm not very worried (at this stage) about this aspect
for Hamster on YARN.
Given your above stat, I'm even more confident Hamster should be fine presently.
bq. the requirement that we launch one container at a time (and copy binaries
et al), will be difficult to overcome
I'm not sure I follow. There is no requirement to launch one container at a
time, how did that come up? An ApplicationMaster can, and should, launch
multiple containers via threads etc. The MR AppMaster does that already.
----
bq. the really big clusters do this with sophisticated network design (limiting
the number of nodes served by each NFS server, multiple subnets) and
TCP-over-IB protocols running at ~50Gbps, so systems connected via lesser
networks can't compete
That's fundamentally the reason Hadoop MapReduce Classic and YARN use HDFS to
make the system more scalable/easy-to-manage with the other characteristics it
brings along.
----
bq. please feel free to consider adding node-to-node collective communications
In the YARN design, the wireup is facilitate via the application's AppMaster.
Thus each worker process just needs to share it's info with it's AppMaster for
wireup. Again, the MR AM already does this for MR jobs.
----
Overall, as disappointed I am, I can understand that your priorities lie
elsewhere.
OTOH, I'd really appreciate if you could share your still-born code or even a
prototype. I'm pretty sure there will be volunteers to take it forward since I
know of sufficient interest in running MPI on YARN. Thanks!
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI
> (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the
> past, running MPI application on a Hadoop cluster was achieved using Hadoop
> Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was
> kludgy. After the resource-manager separation from JobTracker in Hadoop, we
> have all the tools needed to make MPI a first-class citizen on a Hadoop
> cluster. I am currently working on the patch to make MPI an
> application-master. Initial version of this patch will be available soon
> (hopefully before September 10.) This jira will track the development of
> Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
