[
https://issues.apache.org/jira/browse/MAPREDUCE-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13702203#comment-13702203
]
Chris Nauroth commented on MAPREDUCE-5278:
------------------------------------------
{code}
public void testJTStagingOnNondefaultFS() throws Exception {
MiniDFSCluster dfs = null;
MiniMRCluster mr = null;
try {
dfs = new MiniDFSCluster(conf, 1, true, null);
FileSystem fileSys = dfs.getFileSystem();
mr = new MiniMRCluster(1, fileSys.getUri().toString(), 1);
runWithConfJTStagingOnNondefaultFS(mr.createJobConf());
} finally {
mr.shutdown();
dfs.shutdown();
}
}
{code}
In the finally block, could you please check that mr and dfs are not null
before calling shutdown on each one? The constructors can throw exceptions, so
it's possible that they weren't initialized.
Aside from that, it appears that all of the feedback from Bikas has been
addressed. I think we'll be ready to commit once this last part has been
fixed. Thanks, Xi!
> Distributed cache is broken when JT staging dir is not on the default FS
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-5278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distributed-cache
> Affects Versions: 1-win
> Environment: Windows
> Reporter: Xi Fang
> Assignee: Xi Fang
> Fix For: 1-win
>
> Attachments: MAPREDUCE-5278.2.patch, MAPREDUCE-5278.3.patch,
> MAPREDUCE-5278.4.patch, MAPREDUCE-5278.patch
>
>
> Today, the JobTracker staging dir ("mapreduce.jobtracker.staging.root.dir) is
> set to point to HDFS, even though other file systems (e.g. Amazon S3 file
> system and Windows ASV file system) are the default file systems.
> For ASV, this config was chosen and there are a few reasons why:
> 1. To prevent leak of the storage account credentials to the user's storage
> account;
> 2. It uses HDFS for the transient job files what is good for two reasons – a)
> it does not flood the user's storage account with irrelevant data/files b) it
> leverages HDFS locality for small files
> However, this approach conflicts with how distributed cache caching works,
> completely negating the feature's functionality.
> When files are added to the distributed cache (thru files/achieves/libjars
> hadoop generic options), they are copied to the job tracker staging dir only
> if they reside on a file system different that the jobtracker's. Later on,
> this path is used as a "key" to cache the files locally on the tasktracker's
> machine, and avoid localization (download/unzip) of the distributed cache
> files if they are already localized.
> In this configuration the caching is completely disabled and we always end up
> copying dist cache files to the job tracker's staging dir first and
> localizing them on the task tracker machine second.
> This is especially not good for Oozie scenarios as Oozie uses dist cache to
> populate Hive/Pig jars throughout the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
