[ https://issues.apache.org/jira/browse/HADOOP-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-7016: ----------------------------------- Assignee: Rajesh Balamohan Status: Patch Available (was: Open) > Optimize locking codepath in LocalDirAllocator.getLocalPathToRead() and > reduce creating many Path objects > --------------------------------------------------------------------------------------------------------- > > Key: HADOOP-7016 > URL: https://issues.apache.org/jira/browse/HADOOP-7016 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 0.20.2 > Reporter: Rajesh Balamohan > Assignee: Rajesh Balamohan > Fix For: 0.20.2 > > Attachments: HADOOP-7016-v2.Yahoo.patch, HADOOP-7016-v3.Yahoo.patch, > HADOOP-7016-Yahoo.patch > > > LocalDirAllocator.getLocalPathToRead() is called from TaskTracker to retrieve > temporary map output location. This method is synchronized and amount of time > spent on this method directly affects the performance of TaskTracker's > ability to serve faster. This JIRA is created to reduce the additional fat > in getLocalPathToRead() and reduce creating the number of Path obejcts which > are expensive. > 1. getLocalPathToRead() internally calls RawLocalFileSystem.exists() in order > to check the existence of a local file. RawLocalFileSystem.exists() > internally creates a FileStatus object which needs to be populated with > lastModified, length, directory details etc. This might not be necessary for > just checking file existence. > 2. Creating a Path object out of a string requires fair amount of processing. > It might be worth to store "localDirsPath" in LocalDirAllocator instead of > storing "localDirs". This would avoid the number of times Path() object is > created in LocalDirAllocator. > Any other codepath using LocalDirAllocator would also benefit from the above > 2 optimizations. > The attached patch addresses the above 2 issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org