[
https://issues.apache.org/jira/browse/MAPREDUCE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192346#comment-14192346
]
Zhijie Shen commented on MAPREDUCE-6052:
----------------------------------------
The call chain is: copyAndConfigureFiles(2 args) -> copyAndConfigureFiles(3
args) -> addLog4jToDistributedCache -> copyLog4jPropertyFile
1. In copyAndConfigureFiles, the following code has bee executed before. We
don't need to invoke them again in addLog4jToDistributedCache. and we can
combine addLog4jToDistributedCache and copyLog4jPropertyFile into one method.
{code}
// Set the working directory
if (job.getWorkingDirectory() == null) {
job.setWorkingDirectory(jtFs.getWorkingDirectory());
}
{code}
{code}
short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10);
{code}
2. The same bunch of code is already executed before in copyAndConfigureFiles(3
args). No need to repeat in copyLog4jPropertyFile, and mapredSysPerms can be
passed as an arg.
{code}
LOG.debug("default FileSystem: " + jtFs.getUri());
FsPermission mapredSysPerms =
new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION);
if (!jtFs.exists(submitJobDir)) {
throw new IOException("Cannot find job submission directory! "
+ "It should just be created, so something wrong here.");
}
{code}
3. {{FileSystem localFs = FileSystem.getLocal(conf);}} only make sense to the
if condition is true.
{code}
FileSystem localFs = FileSystem.getLocal(conf);
if (pathURI.getScheme() == null) {
//default to the local file system
//check if the file exists or not first
if (!localFs.exists(path)) {
throw new FileNotFoundException("File " + file + " does not exist.");
}
finalPath = path.makeQualified(localFs.getUri(),
localFs.getWorkingDirectory()).toString();
}
else {
// check if the file exists in this file system
// we need to recreate this filesystem object to copy
// these files to the file system ResourceManager is running
// on.
FileSystem fs = path.getFileSystem(conf);
if (!fs.exists(path)) {
throw new FileNotFoundException("File " + file + " does not exist.");
}
finalPath = path.makeQualified(fs.getUri(),
fs.getWorkingDirectory()).toString();
}
{code}
I still have one question about a corner case:
1. Say if we set
mapreduce.job.log4j-properties-file=blah/blah/container-log4j.properties, we're
going to add the param {{-Dlog4j.configuration=container-log4j.properties}},
right? So are we going to use the default {{container-log4j.properties}} or the
uploaded one? Will the default container-log4j.properties be overwritten?
> Support overriding log4j.properties per job
> -------------------------------------------
>
> Key: MAPREDUCE-6052
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6052
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.5.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-6052-v2.patch, MAPREDUCE-6052-v3.patch,
> MAPREDUCE-6052-v4.patch, MAPREDUCE-6052.patch
>
>
> For current MR application, the "log4j.configuration" is hard coded to
> container-log4j.properties within each node. We still need flexibility to
> override it per job like what we do in MRV1.
> {code}
> public static void addLog4jSystemProperties(
> String logLevel, long logSize, int numBackups, List<String> vargs) {
> vargs.add("-Dlog4j.configuration=container-log4j.properties");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
