Hello Mohit,
I am looking at some hadoop tuning parameters like io.sort.mb,
mapred.child.javaopts etc.
- My question was where to look at for current setting
The default settings as well as the documentations can be found in Hadoop
directory:
src/mapred/mapred-default.xml
src/core/core-default.xml
src/hdfs/hdfs-default.xml
- Are these settings configured cluster wide or per job?
Some settings are configured cluster wide, e.g. the number of map/reduce
slots per node, while some settings are configured per job, e.g.
io.sort.mb. It depends on the functionality of that specific parameter.
- What's the best way to look at reasons of slow performance?
Well, I want to introduce Starfish to you. Starfish is a self-tuning system
built on Hadoop to provide good performance automatically, without any need
for users to understand and manipulate the many tuning knobs in Hadoop.
With Starfish, you can analyze the performance of your Hadoop job at fine
grained level, e.g. the time for map processing, spilling, merging,
shuffling, sorting, and reduce processing. So you can understand which
part is the bottleneck of the performance.
You can also ask what-if questions, e.g. What if I double io.sort.mb ?,
and Starfish will predict the new behaviour of the job, so you can better
understand how these parameters work. In addition, you can simply delegate
Starfish to find the optimal configurations for you to achieve the best
performance.
Welcome to join our Google Group to discuss more about Starfish and any
feedback will be appreciated. If you meet any problems, please don't
hesitate to let us know. The Group address is
http://groups.google.com/group/hadoop-starfish.
Thanks,
Jie
-
Starfish Group, Duke University
Starfish Homepage: www.cs.duke.edu/starfish/
Starfish Google Group: http://groups.google.com/group/hadoop-starfish
