You also can increase the memory using command line parameter. <JAR FILE> -D mapred.child.java.opts=-Xmx2048M <INPUT_PARAMETER>
On Sat, Jul 28, 2012 at 8:30 AM, Pat Ferrel <[email protected]> wrote: > I've been changing the hadoop/conf/mapred-site.xml : > > <property> > <name>mapred.child.java.opts</**name> > <value>-Xmx2048m</value> > <description>map heap size for child task</description> > </property> > > This ups the task heap to 2G. > > > On 7/26/12 7:12 PM, Lance Norskog wrote: > >> Increase the memory size or split the file! >> >> On Thu, Jul 26, 2012 at 5:37 AM, pricila rr <[email protected]> wrote: >> >>> I'm trying to transform a file .txt of 1gb for seqfile and the error >>> occurs: OutOfMemoryError: Java heap space >>> How to solve? >>> I am using Hadoop and Mahout. >>> >>> $MAHOUT_HOME/bin/mahout seqdirectory --input '/home/usuario/Área de >>> Trabalho/Dados/base1.txt' --output '/home/usuario/Área de >>> Trabalho/seqFile/base1File' -c UTF-8 >>> MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. >>> Warning: $HADOOP_HOME is deprecated. >>> >>> Running on hadoop, using /home/usuario/hadoop/bin/**hadoop and >>> HADOOP_CONF_DIR=/home/usuario/**hadoop/conf >>> MAHOUT-JOB: >>> /home/usuario/trunk/examples/**target/mahout-examples-0.8-** >>> SNAPSHOT-job.jar >>> Warning: $HADOOP_HOME is deprecated. >>> >>> 12/07/26 09:18:28 INFO common.AbstractJob: Command line arguments: >>> {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], >>> --fileFilterClass=[org.apache.**mahout.text.**PrefixAdditionFilter], >>> --input=[/home/usuario/Área de Trabalho/Dados/base1.txt], --keyPrefix=[], >>> --output=[/home/usuario/Área de Trabalho/seqFile/base1File], >>> --startPhase=[0], --tempDir=[temp]} >>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space >>> at java.util.Arrays.copyOf(**Arrays.java:2882) >>> at >>> java.lang.**AbstractStringBuilder.**expandCapacity(** >>> AbstractStringBuilder.java:**100) >>> at java.lang.**AbstractStringBuilder.append(** >>> AbstractStringBuilder.java:**390) >>> at java.lang.StringBuilder.**append(StringBuilder.java:119) >>> at >>> org.apache.mahout.text.**PrefixAdditionFilter.process(** >>> PrefixAdditionFilter.java:62) >>> at >>> org.apache.mahout.text.**SequenceFilesFromDirectoryFilt**er.accept(** >>> SequenceFilesFromDirectoryFilt**er.java:90) >>> at org.apache.hadoop.fs.**FileSystem.listStatus(**FileSystem.java:845) >>> at org.apache.hadoop.fs.**FileSystem.listStatus(**FileSystem.java:867) >>> at >>> org.apache.mahout.text.**SequenceFilesFromDirectory.**run(** >>> SequenceFilesFromDirectory.**java:98) >>> at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:65) >>> at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:79) >>> at >>> org.apache.mahout.text.**SequenceFilesFromDirectory.**main(** >>> SequenceFilesFromDirectory.**java:53) >>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method) >>> at >>> sun.reflect.**NativeMethodAccessorImpl.**invoke(** >>> NativeMethodAccessorImpl.java:**39) >>> at >>> sun.reflect.**DelegatingMethodAccessorImpl.**invoke(** >>> DelegatingMethodAccessorImpl.**java:25) >>> at java.lang.reflect.Method.**invoke(Method.java:597) >>> at >>> org.apache.hadoop.util.**ProgramDriver$**ProgramDescription.invoke(** >>> ProgramDriver.java:68) >>> at org.apache.hadoop.util.**ProgramDriver.driver(** >>> ProgramDriver.java:139) >>> at org.apache.mahout.driver.**MahoutDriver.main(**MahoutDriver.java:195) >>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method) >>> at >>> sun.reflect.**NativeMethodAccessorImpl.**invoke(** >>> NativeMethodAccessorImpl.java:**39) >>> at >>> sun.reflect.**DelegatingMethodAccessorImpl.**invoke(** >>> DelegatingMethodAccessorImpl.**java:25) >>> at java.lang.reflect.Method.**invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:156) >>> >> >> >> >
