[
https://issues.apache.org/jira/browse/HADOOP-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-7016:
-----------------------------------
Assignee: Rajesh Balamohan
Status: Patch Available (was: Open)
> Optimize locking codepath in LocalDirAllocator.getLocalPathToRead() and
> reduce creating many Path objects
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7016
> URL: https://issues.apache.org/jira/browse/HADOOP-7016
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Fix For: 0.20.2
>
> Attachments: HADOOP-7016-v2.Yahoo.patch, HADOOP-7016-v3.Yahoo.patch,
> HADOOP-7016-Yahoo.patch
>
>
> LocalDirAllocator.getLocalPathToRead() is called from TaskTracker to retrieve
> temporary map output location. This method is synchronized and amount of time
> spent on this method directly affects the performance of TaskTracker's
> ability to serve faster. This JIRA is created to reduce the additional fat
> in getLocalPathToRead() and reduce creating the number of Path obejcts which
> are expensive.
> 1. getLocalPathToRead() internally calls RawLocalFileSystem.exists() in order
> to check the existence of a local file. RawLocalFileSystem.exists()
> internally creates a FileStatus object which needs to be populated with
> lastModified, length, directory details etc. This might not be necessary for
> just checking file existence.
> 2. Creating a Path object out of a string requires fair amount of processing.
> It might be worth to store "localDirsPath" in LocalDirAllocator instead of
> storing "localDirs". This would avoid the number of times Path() object is
> created in LocalDirAllocator.
> Any other codepath using LocalDirAllocator would also benefit from the above
> 2 optimizations.
> The attached patch addresses the above 2 issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
