[
https://issues.apache.org/jira/browse/HADOOP-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587296#action_12587296
]
Chris Douglas commented on HADOOP-3203:
---------------------------------------
bq. Using listStatus(Path) makes only one RPC. Do you think we should go for
FileSystem.getFileStatus(Path) and make two RPCs?
This section of code should be called once per TaskTracker per Job, right? The
difference between one RPC and two RPCs here is insufficient to measure, let
alone effect a performance bottleneck or load we should worry about. Since the
5x expansion of the jar is a pessimistic estimate, the config should scarcely
matter, but given HADOOP-3048 and the possibility of serialized objects in the
config, it's worth checking its size, too.
> TaskTracker::localizeJob doesn't provide the correct size to LocalDirAllocator
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3203
> URL: https://issues.apache.org/jira/browse/HADOOP-3203
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.15.3
> Reporter: Chris Douglas
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3203.txt
>
>
> In TaskTracker::localizeJob:
> {code}
> // Get sizes of JobFile and JarFile
> // sizes are -1 if they are not present.
> FileSystem fileSystem = FileSystem.get(fConf);
> FileStatus status[] = fileSystem.listStatus(new
> Path(jobFile).getParent());
> long jarFileSize = -1;
> long jobFileSize = -1;
> for(FileStatus stat : status) {
> if (stat.getPath().toString().contains("job.xml")) {
> jobFileSize = stat.getLen();
> } else {
> jobFileSize = -1;
> }
> if (stat.getPath().toString().contains("job.jar")) {
> jarFileSize = stat.getLen();
> } else {
> jarFileSize = -1;
> }
> }
> {code}
> One or both of jobFileSize and jarFileSize will be -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
