Ben Podgursky created MAPREDUCE-6450:
----------------------------------------
Summary: mapreduce.job.log4j-properties-file does not work for
files on classpath
Key: MAPREDUCE-6450
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6450
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ben Podgursky
Priority: Minor
JobSubmitter.addLog4jToDistributedCache checks whether
MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE is set, and tries to put it in
the distributed cache if it is set:
{code}
String log4jPropertyFile =
conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
if (!log4jPropertyFile.isEmpty()) {
short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10);
copyLog4jPropertyFile(job, jobSubmitDir, replication);
{code}
This throws an exception inside JobSubmitter.validateFilePath if the file does
not exist on the local filesystem, but is instead on the classpath. This is
unnecessarily restrictive, because later on in MRApps.addLog4jSystemProperties,
the actual configuration of -Dlog4j.configuration works fine if the file is on
the classpath and not a physical file:
{code}
String log4jPropertyFile =
conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, "");
if (log4jPropertyFile.isEmpty()) {
vargs.add("-Dlog4j.configuration=container-log4j.properties");
{code}
(the default file container-log4j.properties is on the classpath anyway)
This is a nuisance because we have a bunch of projects and it's much easier to
distribute shared configuration files via jars and dependencies than raw files.
It's not a blocker because I'm able to work around it by setting
-Dlog4j.configuration in yarn.app.mapreduce.am.command-opts, but it's certainly
hackier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
