Hi JJ,
The code responsible for setting options in hadoop commands is
$HADOOP_COMMON_HOME/bin/hadoop itself, which is a bash shell script.
The "jar" command goes through a 'case' statement to code that says:
elif [ "$COMMAND" = "jar" ] ; then
CLASS=org.apache.hadoop.util.RunJar
However, if you examine some of its sibling cases, you see some that have an
additional line like
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
or
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS"
HADOOP_*_OPTS are defined in $HADOOP_CONF_DIR/hadoop-env.sh, where we see that
$HADOOP_CLIENT_OPTS sets -Xmx128m, and $HADOOP_NAMENODE_OPTS sets -Xmx45000m.
So, you could modify the hadoop shellscript or the hadoop-env.sh file to cause
the desired -Xmx command to be appended to HADOOP_OPTS. I've done this
successfully.
Alternatively, there appears to be a better way available which I have not
used: In $HADOOP_COMMON_HOME/bin/hadoop-config.sh (which gets called during
any hadoop command invocation), there is code which says
# check envvars which might override default args
if [ "$HADOOP_HEAPSIZE" != "" ]; then
#echo "run with heapsize $HADOOP_HEAPSIZE"
JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
#echo $JAVA_HEAP_MAX
fi
This implies that doing
export HADOOP_HEAPSIZE=1000
in the shell before invoking the "hadoop" command, would cause a 1GB heapsize
to be set via -Xmx.
Hope this helps,
--Matt
On Apr 28, 2011, at 5:22 PM, Mapred Learn wrote:
Hi Soren/Bobby,
Thanks for the response !
This is not a map-red but just a jar that runs on Hadoop Client to get/put data
on HDFs but I want the heap for this JVM to be more than 1 GB.
What you guys suggested is I guess for mapper/reducer code.
Thanks,
On Thu, Apr 28, 2011 at 2:41 PM, Soren Flexner
<sflex...@gmail.com<mailto:sflex...@gmail.com>> wrote:
Hi JJ,
I think the property you're looking for is mapred.child.java.opts
${HADOOP_HOME}/conf/mapred-site.xml:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m</value>
.....etc
For example.
Or
hadoop --config <> jar <jarname> <main class> -D
mapred.child.java.opts=-Xmx1024M
-s
On Thu, Apr 28, 2011 at 2:26 PM, Mapred Learn
<mapred.le...@gmail.com<mailto:mapred.le...@gmail.com>> wrote:
Hi,
I am runnnig a hadoop jar command as:
hadoop --config <> jar <jarname> <main class> -conf <conf file>
My question is how and where can I specify -Xmx option to increase heap
assigned to my JVM ?
Thanks in advance
-JJ
