[
https://issues.apache.org/jira/browse/MAPREDUCE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978356#comment-13978356
]
Jason Lowe commented on MAPREDUCE-5812:
---------------------------------------
Yes, the findbug warnings need to be addressed, otherwise someone calling
OutputCommitter.isRecoveryEnabled with a reference that is typed as a mapreduce
JobContext but is really a mapred JobContext would end up calling the default
implementation of mapreduce OutputCommitter#isRecoveryEnabled which will do the
wrong thing unless the derived committer remembered to override that as well.
Adding a bridging function in mapred OutputCommitter as I mentioned earlier
should do the trick, and that's what the other methods are already doing. See
the public final methods towards the bottom of mapred OutputCommitter. For
example we need to add something like this:
{code}
/**
* This method implements the new interface by calling the old method. Note
* that the input types are different between the new and old apis and this
* is a bridge between the two.
*/
@Override
public final
boolean isRecoverySupported(org.apache.hadoop.mapreduce.JobContext context)
throws IOException {
return isRecoverySupported((JobContext) context);
}
{code}
This would also allow MRAppMaster to simply call
committer.isRecoveryEnabled(jobContext) after the proper JobContext is created
without the mapred OutputCommitter casting, e.g.:
{code}
private boolean isRecoverySupported(OutputCommitter committer2)
throws IOException {
boolean isSupported = false;
if (committer != null) {
JobContext jobContext;
if (newApiCommitter) {
jobContext = new JobContextImpl(getConfig(),
TypeConverter.fromYarn(getJobId()));
} else {
jobContext = new org.apache.hadoop.mapred.JobContextImpl(
new JobConf(getConfig()), TypeConverter.fromYarn(getJobId()));
}
isSupported = committer.isRecoverySupported(jobContext);
}
return isSupported;
}
{code}
> Make task context available to OutputCommitter.isRecoverySupported()
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-5812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5812
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.3.0
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Attachments: MAPREDUCE-5812.1.patch, MAPREDUCE-5812.2.patch,
> MAPREDUCE-5812.3.patch, MAPREDUCE-5812.4.patch
>
>
> Background
> ==========
> The system like Hive provides its version of OutputCommitter. The custom
> implementation of isRecoverySupported() requires task context. From
> taskContext:getConfiguration(), hive checks if hive-defined specific
> property is set or not. Based on the property value, it returns true or
> false. However, in the current OutputCommitter:isRecoverySupported(), there
> is no way of getting task config. As a result, user can't turn on/off the
> MRAM recovery feature.
> Proposed resolution:
> ===============
> 1. Pass Task Context into isRecoverySupported() method.
> Pros: Easy and clean
> Cons: Possible backward compatibility issue due to aPI changes. (Is it true?)
> 2. Call outputCommitter.setupTask(taskContext) from MRAM: The new
> OutputCommitter will store the context in the class level variable and use it
> from isRecoverySupported()
> Props: No API changes. No backward compatibility issue. This call can be made
> from MRAppMaster.getOutputCommitter() method for old API case.
> Cons: Might not be very clean solution due to class level variable.
> Please give your comments.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
