[
https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251919#comment-13251919
]
Steve Loughran commented on MAPREDUCE-2911:
-------------------------------------------
Classic HPC systems contain some expections that aren't present in normal
Hadoop clusters
# low-latency IPC & APIs to access it for messaging
# shared location-independent filesystem (e.g. SAN, GPFS, ...)
# highly calibrated C++ compiler and library systems to guarantee consistent
output of FPU- and GPU- intensive code.
# C/C++ code with lots of MPI code scattered through out for synchronization.
# either checkpointing of long-lived work or timespan limits for jobs.
There are some other features that aren't necessarily design goals but have
arisen
# lots of smaller per-facility sites with different architectures rather than
one or two large-scale datacentres.
# limited local storage (due to cost of storage solution)
# a reliance on zero-costed internet-2/SuperJanet/... interconnect to pull data
from shared repositories (for CERN, the national tier-1 sites such as
Rutherford Appleton Laboratories)
# high operations cost relative to storage and compute capacity (due to small,
heterogenous, multi-site clusters, selected storage solutions)
If you look at the UK university grid
[http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html] you can see that although
there are lots of clusters, they are of limited storage capacity -that storage
also forces you to choose where to run the work or rely on job preheating to
pull it in from RAL or elsewhere. (latency to do this is lower than pulling off
tape). You can also see that there are lot of jobs in the queues, including
short-lived health tests that verify work reaches the expected answers. I don't
know about the duration/needs of the actual work jobs.
When you consider job startup delays you have to look at time to fetch data
over long-haul connections, maybe compile code for target cluster, and
recognise that without a SAN you can't expect uniform access times to all data.
What you would get from MPI over hadoop is the ability to run MPI work on the
cluster -a cluster which, if it also had infiniband on, would have low-latency
interconnections. (yes, there is a cost for that, but you may want it for a
shared cluster).
What about an MPI mechanism that has a Grid Scheduler that block-rents a set of
machines that an then be used for multiple jobs off the MPI queue, and which
aren't released after each job? Once the capacity on the hosts is allocated,
health checks can verify the machines work properly, then it can await work.
The scheduler can look at the pending queue and flex its set of machines based
on expected load?
Job startup would be reduce to the time to push out work to the pre-allocated
hosts, which doesn't need to rely on heartbeats and could use Zookeeper or
other co-ordination services.
This wouldn't be a drop in replacement for one of the big supercomputing
clusters, but it would let people run MPI jobs within a Hadoop cluster.
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI
> (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the
> past, running MPI application on a Hadoop cluster was achieved using Hadoop
> Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was
> kludgy. After the resource-manager separation from JobTracker in Hadoop, we
> have all the tools needed to make MPI a first-class citizen on a Hadoop
> cluster. I am currently working on the patch to make MPI an
> application-master. Initial version of this patch will be available soon
> (hopefully before September 10.) This jira will track the development of
> Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
