Submitting job information via DFS in Map/Reduce causing consistency and
performance issues
-------------------------------------------------------------------------------------------
Key: HADOOP-5350
URL: https://issues.apache.org/jira/browse/HADOOP-5350
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Hemanth Yamijala
Job submission involves two steps: submitting jobs to the System directory on
DFS (done by the client), then submit the job via the JobSubmissionProtocol to
JobTracker. This two step process is seen to have some issues:
- Since the files need to be read from DFS, slowness in the DFS can cause job
initialization to become costly. We faced this as described in HADOOP-5286 and
HADOOP-4664.
- The two step process could lead to inconsistent information being left around
- like in HADOOP-5327 and HADOOP-5335.
This JIRA is to explore options to remove the two step process in submitting a
job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
