[
https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251799#comment-13251799
]
Arun C Murthy commented on MAPREDUCE-2911:
------------------------------------------
Ralph, thanks for the heads up. It's disappointing it's some way away...
Regarding your observations, but no offense taken, some clarifications:
Foremost, there are a couple of reasons why the container launch on YARN should
be more than sufficient:
# People using MPI on Hadoop are generally running very heavy computation
models which run for several minutes at least if not for several hours. In
fact, I've seen lots of MPI-like jobs run on Hadoop MapReduce which run for
several days! In such cases a few seconds is really pure noise. (Also, we've
added several features to CapacityScheduler to actually support such apps in
hadoop-1.x.)
# Hadoop clusters are also, typically, heavily over-subscribed which means
tasks (containers) spend a lot of time in queue waiting for resources to be
available.
Having said that, there is no doubt we are focussed on improving YARN. It's
currently significantly faster than MapReduce Classic (JobTracker/TaskTracker)
in hadoop-1.x for container (task) launch etc. and we have more improvements up
our sleeve for the future.
----
A very important factor for container launch in YARN (as with current MR) is
that the system itself takes responsibility for 'setting up' the container by
copying it's jars/shared-objects/binaries from HDFS. An easy way to speed up
container launch is to pre-distributed MPI binaries on all nodes to do a
perf-comparision. Is this something you've already tried?
In contrast you probably, already, did that on Torque/Moab/Maui since they have
to be distributed before the MPI 'job' is launched?
----
Regarding 'collective communication support' can you please add more colour?
With my naive understanding of the MPI world, I don't see why the same
mechanisms can't be used in YARN.
YARN doesn't mandate application communication mechanisms and just leaves it up
to the applications. So, I fail to see why this comes up for MPI applications -
happy to be educated! *smile*
----
Regarding your current area of focus i.e. running Hadoop MapReduce (and HDFS?)
on Torque etc., this is something we in the Hadoop community have already tried
(and abandoned) with "Hadoop On Demand" i.e. HoD system (HoD jira: HADOOP-1301.
HoD removed via HADOOP-7137). There are several reasons why we went away, and
too much water under the bridge.
----
Overall, I'm sure there are several people in Hadoop community who would love
to use MPI on Hadoop/YARN, they already do that by bending Hadoop MapReduce in
myriad & scary ways! I'm sure Milind, like me, has seen several examples!
*smile*
I can see why some of the issues you've brought up (milisec latency etc.)
matter to folks in MPI world, in some ways it's a clash of the Hadoop and HPC
worlds and the fallout is reasonable & expected! *smile*
However, I still believe Hamster, as envisaged here, has a lot of value to the
Hadoop community, if you choose to focus on it. Thanks!
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI
> (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the
> past, running MPI application on a Hadoop cluster was achieved using Hadoop
> Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was
> kludgy. After the resource-manager separation from JobTracker in Hadoop, we
> have all the tools needed to make MPI a first-class citizen on a Hadoop
> cluster. I am currently working on the patch to make MPI an
> application-master. Initial version of this patch will be available soon
> (hopefully before September 10.) This jira will track the development of
> Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
