[
https://issues.apache.org/jira/browse/MAPREDUCE-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675706#comment-13675706
]
Sandy Ryza commented on MAPREDUCE-5184:
---------------------------------------
[~zjshen], thanks a ton for writing this up. Overall, it looks great to me. I
have a bunch of stylistic/grammatical suggestions. (Obviously) feel free to
take what sounds good to you and leave out what does not. For the sake of
brevity, I've mostly written out changes that would sound the best to me
without explanations, but I would be happy to provide the rationale on any of
them.
In general, other docs have tended to use the all-capitalized YARN over Yarn,
so I would advocate for following this convention.
{code}
+ MapReduce NextGen (aka MRv2) has spin off the resource management
+ responsibility, which has upgraded to Yarn, a general-purpose, distributed,
+ application management framework. Meanwhile, MapReduce remains as a pure
+ distributed computation framework.
{code}
MapReduce NextGen (aka MRv2) has spun off resource management capabilities into
YARN, a general purpose, distributed application management framework.
MapReduce remains as a pure distributed computation framework.
{code}
+ In general, the previous MapReduce framework (aka MRv1) has been reused and
+ no major surgery has been conducted on it. Therefore, MRv2 is able to ensure
{code}
In general, the APIs of the previous MapReduce framework (aka MRv1) have been
preserved without the need for any major surgery.
{code}
+ satisfactory compatibility to MRv1 applications. However, because of
+ improvement and code refactoring, backward compatibility of some APIs still
+ are broken.
{code}
However, due to some improvements and code refactorings, a few APIs have been
rendered backward-incompatible.
{code}
+ First of all, we ensure binary compatibility to the applications that use
+ "mapred" APIs. It means that the applications which are compiled with MRv1
+ "mapred" APIs can directly run on Yarn, while recompiling is not required.
{code}
First, we ensure binary compatibility for applications that use the old
"mapred" APIs. This means that applications which were compiled with MRv1
"mapred" APIs can run directly on YARN without recompilation.
{code}
+* {Source Compatibility}
+
+ However, we cannot ensure binary compatibility to the applications that use
+ "mapreduce" APIs, as these APIs have evolved a lot since MRv1. Instead, the
+ applications only need to be recompiled against MRv2 "mapreduce" APIs. The
+ important binary incompatible spots are Counter and CounterGroup, Users are
+ recommended to recompile their applications if they used the two interfaces.
{code}
Not sure that it makes sense to have this section and its predecessor described
as "Source Compatibility" vs. "Binary Compatibility". MAPREDUCE-4942, for
example, tracks changes that are binary-compatible, but not source-compatible.
Maybe it would make more sense to demarcate them as old API vs. new API or
incompatilities vs. compatibilities.
{code}
+ MRAdmin is removed in MRv2. Since it is supposed to be used through CLI
+ commands, we don't support binary compatibility for the applications that use
+ this class directly.
{code}
MRAdmin has been removed in MRv2 because because the mradmin commands no longer
exist. They have been replaced by the commands in rmadmin.
{code}
+* {Tradeoff between MRv1 Users and Early MRv2 Adopters}
{code}
Should be tradeoff*s*
{code}
+ Unfortunately, some changes to "mapred" APIs lead to binary incompatible
issues
+ for the early MRv2 adopters, in particular Hadoop 0.23 users. They are caused
+ by the incompatible method signature. For example, ProgramDriver#drive
returns
+ void in MRv1, but returns int in MRv2. Under the either-or situation, we
choose
+ to be compatible to MRv1 applications, which have a larger user base. Bellow
is
+ the list of the spots where Hadoop 0.23 applications will be broken.
{code}
Unfortunately, maintaining compatibility with the "mapred" APIs in MRv1 may led
to binary incompatibility issues for early MRv2 adopters, in particular Hadoop
0.23 users. For the "mapred" APIs, we have chosen to be compatible with MRv1
applications, which have a larger user base. Below is the list of MapReduce
APIs which are incompatible with Hadoop 0.23.
{code}
+* {Malicious}
+
+ For the users who are going to try hadoop-examples-1.x.x.jar on Yarn, please
+ note that "hadoop -jar hadoop-examples-1.x.x.jar" is still using the
2.x.x.jar.
+ Users should either remove hadoop-mapreduce-examples-2.x.x.jar from the
+ classpath or set "HADOOP_USER_CLASSPATH_FIRST=true" and
+ "HADOOP_CLASSPATH=...:hadoop-examples-1.x.x.jar" to run their target examples
+ jar.
{code}
I'm a little confused by this. What does it mean that "hadoop -jar
hadoop-examples-1.x.x.jar" is still using the 2.x.x.jar? Where is the 2.x.x
jar coming from?
> Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-5184
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5184
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: documentation
> Reporter: Arun C Murthy
> Assignee: Zhijie Shen
> Fix For: 2.1.0-beta
>
> Attachments: MAPREDUCE-5184.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
