I've run into a situation where it would be helpful to set specific
configuration variables local to a data/task node. I've got a solution, but
I'm curious if there is a best practice around this and if I'm doing it in a
reasonable way.
Basically what we've got is a number of machines that have 2 Cores/4GB of
memory. For those boxes we have some options configured higher than the
default in mapred-site.xml ( specifically mapred.child.java.opts set to
-Xmx1024m ). We recently added a few additional boxes that have 2 Cores/2GB of
memory and the mapred.child.java.opts causes those boxes to swap so we'd like
to configure those boxes to set mapred.child.java.opts to -Xmx512m. What I
found is that if on the data/task node that if I change the value in
mapred-site.xml it is overridden, but if I set the parameter using
<final>true</final> it is used. So effectively now I've got a different
mapred-site.xml on each of the nodes.
My question is, is this a reasonable way of going about this? Is there a
best practice for dealing with minor node-specific configuration differences?
What I'd really want is not only a mapred-site.xml, but a mapred-node.xml as
well that has node specific overrides. What I can't tell is if I make all the
site level configuration changes in the name node mapred-site.xml and then the
node specific mapred-site.xml files to node local changes if that does what I'm
looking for.
Any insight would be appreciated.
Andy
