if the 'mapred.system.dir' in the client jobconf is different from the
JobTracker's value job submission fails
--------------------------------------------------------------------------------------------------------------
Key: HADOOP-3135
URL: https://issues.apache.org/jira/browse/HADOOP-3135
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.16.1
Environment: all
Reporter: Alejandro Abdelnur
Priority: Blocker
Fix For: 0.17.0
Until Hadoop 0.13 or so, at submission time the full path of the job.xml and
all supporting files in DFS was given by the client to the jobtracker.
Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID
from the jobtracker and creating the directory for all the supporting files
using the a system-dir computed from the local jobconf.
Line 696-7 in the JobClient:
String jobId = jobSubmitClient.getNewJobId();
Path submitJobDir = new Path(job.getSystemDir(), jobId);
This makes submissions to fail when the value of the 'mapred.system.dir' on the
client is different from the one in the JobTracker.
A simple way o fixing this would be to introduce a new method in the
JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker system
dir and use that dir for uploading all the files on submission.
----
For the future: A more comprehensive way of this doing would to obtain a base
jobConf from the jobtracker, carrying final information for each element and
the construct the job.xml on the client using the final semantics. And, in this
case the 'mapred.system.dir' property should be set as final in the jobtracker.
As there may be some configuration properties that are sensitive and for
security reasons should not be exposed to the clients a new flag 'private'
could be introduced and only properties that don't have the 'private' flag
would be send over from the jobtracker to the jobclient for job.xml resolution.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
