[
https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997654#comment-12997654
]
Joydeep Sen Sarma commented on MAPREDUCE-279:
---------------------------------------------
I have been working on maintaining/enhancing MR for FB's use case for last 6
months or so. Here are a few priority areas for us that are relevant to this
discussion:
# Latency.
This is, imho, #1 priority wrt scheduling. As Scott has already remarked, the
ping-response model is broken. So is preemption as an after-thought. We need to
get small/medium jobs scheduled instantly. Period.
# Scalability.
We have made a number of vital fixes to keep the JT working at our scale -
but we have merely bought some time.
# (wrt. ResourceManager) Open API
By which i mean something like Thrift/PB/Avro. We are, of course, most
comfortable with Thrift and it would be nice (but not critical) if it were
possible to build a Thrift wrapper (even if one was not baked in from scratch).
One thing i have found is that writing Thrift services is a breeze because of
inbuilt service framework. Everything else on the serialization side being
equal - this has been a big win for me personally as a developer (and something
to be considered as other distributed execution frameworks try to use the RM).
# Ability to back-plug into older Hadoop versions
This is related to #3. Unlike many other organizations - we cannot make big
jumps in hadoop revisions anymore. We have too many custom changes and we don't
have a QA department. Unlike in the past, where we could have depended on
Yahoo's QA'ed releases - we don't have that luxury anymore (because we are now
both running software at similar versions - and we can't wait until Yahoo has
deployed/QA'ed new versions before deploying newer upgrades).
If the RM api is open (and satisfactory from design perspective) - we can do
the work in-house to our older version of Hadoop to use it. This is critical
for us (although i am not sure it applies to other users).
I cannot emphasize the urgency around #1. Whether we continue to use Hadoop or
not is predicated on big improvements in latency and efficiency (the latter is
a different topic).
I hope #3 and #4 contribute to the discussion around component architecture. At
our scale - i don't think we can build services using large software that is
tightly integrated. We need too much customization and we can't afford the long
upgrade cycles of such tightly integrated software. Of course, this is specific
to our deployment and the requirements of most other deployments is likely to
be quite different.
----
As a developer - i have found the current JobTracker code totally
unmaintainable - I hope the new version (broken across RM/App-Master) is
better. There are several design points that have struck me as particularly
evil:
# synchronous RPC based architecture: limits concurrency and forces bad
implementation choices (see #2)
# crazy locking: this is just bad implementation for the most part - but i hope
the new design/implementation clearly articulates some principles around the
fundamental data structures and how transactional changes to these data
structures are meant to be accomplished.
# poor data structure maintenance: 99% of the data structures in the JT have a
pattern of a:
a. a primary collection (eg: list of all jobs in the system)
b. several secondary indices/views (list of all runnable jobs from above,
list of all completed jobs etc)
Instead of modeling updates to such collections and related views through a
common entry point - updates to primary and secondary data structures are at
disjoint places throughout the code and make maintenance of code a nightmare.
i can only hope that a big rewrite like this will try to address some of these
issues (others - like hard-wiring to specific (M/R) task types - are already
addressed i presume in the new RM).
my 2c.
> Map-Reduce 2.0
> --------------
>
> Key: MAPREDUCE-279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, tasktracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job,
> user-defined component that manages the application execution.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
