[
https://issues.apache.org/jira/browse/MAPREDUCE-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated MAPREDUCE-1904:
----------------------------------------
Attachment: LocalDirAllocator.JPG
LocalDirAllocator_Monitor.JPG
CPU profiler output of a TaskTracker "without" the patch. Profiler output shows
that ~4% of time being spent on internal methods of
LocalDirAllocator.*.getLocalPathToRead(). Rest of them time is spent on the
method itself. This along with the monitor profiling shows synchonization to be
the bottleneck in getLocalPathToRead().
> Reducing locking contention in TaskTracker.MapOutputServlet's
> LocalDirAllocator
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1904
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1904
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.1
> Reporter: Rajesh Balamohan
> Attachments: LocalDirAllocator.JPG, LocalDirAllocator_Monitor.JPG,
> MAPREDUCE-1904-RC10.patch, MAPREDUCE-1904-trunk.patch, profiler output after
> applying the patch.jpg, TaskTracker- yourkit profiler output .jpg, Thread
> profiler output showing contention.jpg
>
>
> While profiling tasktracker with Sort benchmark, it was observed that threads
> block on LocalDirAllocator.getLocalPathToRead() in order to get the index
> file and temporary map output file.
> As LocalDirAllocator is tied up with ServetContext, only one instance would
> be available per tasktracker httpserver. Given the jobid & mapid,
> LocalDirAllocator retrieves index file path and temporary map output file
> path. getLocalPathToRead() is internally synchronized.
> Introducing a LRUCache for this lookup reduces the contention heavily
> (LRUCache with key =jobid +mapid and value=PATH to the file). Size of the
> LRUCache can be varied based on the environment and I observed a throughput
> improvement in the order of 4-7% with the introduction of LRUCache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
