[
https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587505#action_12587505
]
Arun C Murthy commented on HADOOP-3135:
---------------------------------------
Since you are generating a new patch, I'd appreciate if you could weed out
miscellaneous changes like these which aren't necessary:
{noformat}
Index: src/java/org/apache/hadoop/mapred/JobTracker.java
===================================================================
--- src/java/org/apache/hadoop/mapred/JobTracker.java (revision 643322)
+++ src/java/org/apache/hadoop/mapred/JobTracker.java (working copy)
@@ -37,7 +37,6 @@
import java.util.Properties;
import java.util.Set;
import java.util.TreeMap;
-import java.util.HashMap;
import java.util.TreeSet;
import java.util.Vector;
import java.util.concurrent.LinkedBlockingQueue;
@@ -51,18 +50,18 @@
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.ipc.RemoteException;
+import org.apache.hadoop.ipc.Server;
import org.apache.hadoop.ipc.RPC.VersionMismatch;
-import org.apache.hadoop.ipc.Server;
import org.apache.hadoop.metrics.MetricsContext;
import org.apache.hadoop.metrics.MetricsRecord;
import org.apache.hadoop.metrics.MetricsUtil;
import org.apache.hadoop.metrics.Updater;
import org.apache.hadoop.metrics.jvm.JvmMetrics;
import org.apache.hadoop.net.DNSToSwitchMapping;
+import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.net.NetworkTopology;
+import org.apache.hadoop.net.Node;
import org.apache.hadoop.net.NodeBase;
-import org.apache.hadoop.net.Node;
-import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.net.ScriptBasedMapping;
import org.apache.hadoop.util.HostsFileReader;
import org.apache.hadoop.util.ReflectionUtils;
{noformat}
Oh, we probably should cache the value of the system.dir in
JobClient.getSystemDir rather than do an RPC to get it each time...
> if the 'mapred.system.dir' in the client jobconf is different from the
> JobTracker's value job submission fails
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3135
> URL: https://issues.apache.org/jira/browse/HADOOP-3135
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.1
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Subramaniam Krishnan
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: patch-3135-v3.txt, patch-3135-v4.txt
>
>
> Until Hadoop 0.13 or so, at submission time the full path of the job.xml and
> all supporting files in DFS was given by the client to the jobtracker.
> Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job
> ID from the jobtracker and creating the directory for all the supporting
> files using the a system-dir computed from the local jobconf.
> Line 696-7 in the JobClient:
> String jobId = jobSubmitClient.getNewJobId();
> Path submitJobDir = new Path(job.getSystemDir(), jobId);
> This makes submissions to fail when the value of the 'mapred.system.dir' on
> the client is different from the one in the JobTracker.
> A simple way o fixing this would be to introduce a new method in the
> JobSubmissionProtocol 'getSystemDir()' that would return the jobtracker
> system dir and use that dir for uploading all the files on submission.
> ----
> For the future: A more comprehensive way of this doing would to obtain a base
> jobConf from the jobtracker, carrying final information for each element and
> the construct the job.xml on the client using the final semantics. And, in
> this case the 'mapred.system.dir' property should be set as final in the
> jobtracker. As there may be some configuration properties that are sensitive
> and for security reasons should not be exposed to the clients a new flag
> 'private' could be introduced and only properties that don't have the
> 'private' flag would be send over from the jobtracker to the jobclient for
> job.xml resolution.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
