You can turn on the default profiler using mapred.task.profile,
mapred.task.profile.maps and mapred.task.profile.reduces. I have used
a different profiler but had to modify the Hadoop code. Not sure if there
is a way to pass -agentlib param to the child jvms.
Amar
On Mon, 25 Feb 2008, Alexander Mont wrote:
I am interested in examining a MapReduce execution in order to determine the
amount of time it takes to execute each of the following parts of a MapReduce
job:
- Loading of data onto mappers
- Executing map operation
- Sorting/partitioning of map output
- Loading map output onto reducers
- Executing reduce operation
- Writing output of reducers to data files
Does Hadoop have a built-in mechanism to profile a MapReduce application in
this way, and if not, have other mechanisms to do this been developed?
-Alex Mont
