Even changing the memory settings the error continues. What else can I do? And how can I split the file? With smaller files does not occur error. I'm using Mahout and Hadoop on Linux machines, with one master and two slaves.
Thank you. 2012/7/28 Anandha L Ranganathan <[email protected]> > You also can increase the memory using command line parameter. > > <JAR FILE> -D mapred.child.java.opts=-Xmx2048M <INPUT_PARAMETER> > > On Sat, Jul 28, 2012 at 8:30 AM, Pat Ferrel <[email protected]> wrote: > > > I've been changing the hadoop/conf/mapred-site.xml : > > > > <property> > > <name>mapred.child.java.opts</**name> > > <value>-Xmx2048m</value> > > <description>map heap size for child task</description> > > </property> > > > > This ups the task heap to 2G. > > > > > > On 7/26/12 7:12 PM, Lance Norskog wrote: > > > >> Increase the memory size or split the file! > >> > >> On Thu, Jul 26, 2012 at 5:37 AM, pricila rr <[email protected]> > wrote: > >> > >>> I'm trying to transform a file .txt of 1gb for seqfile and the error > >>> occurs: OutOfMemoryError: Java heap space > >>> How to solve? > >>> I am using Hadoop and Mahout. > >>> > >>> $MAHOUT_HOME/bin/mahout seqdirectory --input '/home/usuario/Área de > >>> Trabalho/Dados/base1.txt' --output '/home/usuario/Área de > >>> Trabalho/seqFile/base1File' -c UTF-8 > >>> MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. > >>> Warning: $HADOOP_HOME is deprecated. > >>> > >>> Running on hadoop, using /home/usuario/hadoop/bin/**hadoop and > >>> HADOOP_CONF_DIR=/home/usuario/**hadoop/conf > >>> MAHOUT-JOB: > >>> /home/usuario/trunk/examples/**target/mahout-examples-0.8-** > >>> SNAPSHOT-job.jar > >>> Warning: $HADOOP_HOME is deprecated. > >>> > >>> 12/07/26 09:18:28 INFO common.AbstractJob: Command line arguments: > >>> {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], > >>> --fileFilterClass=[org.apache.**mahout.text.**PrefixAdditionFilter], > >>> --input=[/home/usuario/Área de Trabalho/Dados/base1.txt], > --keyPrefix=[], > >>> --output=[/home/usuario/Área de Trabalho/seqFile/base1File], > >>> --startPhase=[0], --tempDir=[temp]} > >>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > >>> at java.util.Arrays.copyOf(**Arrays.java:2882) > >>> at > >>> java.lang.**AbstractStringBuilder.**expandCapacity(** > >>> AbstractStringBuilder.java:**100) > >>> at java.lang.**AbstractStringBuilder.append(** > >>> AbstractStringBuilder.java:**390) > >>> at java.lang.StringBuilder.**append(StringBuilder.java:119) > >>> at > >>> org.apache.mahout.text.**PrefixAdditionFilter.process(** > >>> PrefixAdditionFilter.java:62) > >>> at > >>> org.apache.mahout.text.**SequenceFilesFromDirectoryFilt**er.accept(** > >>> SequenceFilesFromDirectoryFilt**er.java:90) > >>> at org.apache.hadoop.fs.**FileSystem.listStatus(**FileSystem.java:845) > >>> at org.apache.hadoop.fs.**FileSystem.listStatus(**FileSystem.java:867) > >>> at > >>> org.apache.mahout.text.**SequenceFilesFromDirectory.**run(** > >>> SequenceFilesFromDirectory.**java:98) > >>> at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:65) > >>> at org.apache.hadoop.util.**ToolRunner.run(ToolRunner.**java:79) > >>> at > >>> org.apache.mahout.text.**SequenceFilesFromDirectory.**main(** > >>> SequenceFilesFromDirectory.**java:53) > >>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method) > >>> at > >>> sun.reflect.**NativeMethodAccessorImpl.**invoke(** > >>> NativeMethodAccessorImpl.java:**39) > >>> at > >>> sun.reflect.**DelegatingMethodAccessorImpl.**invoke(** > >>> DelegatingMethodAccessorImpl.**java:25) > >>> at java.lang.reflect.Method.**invoke(Method.java:597) > >>> at > >>> org.apache.hadoop.util.**ProgramDriver$**ProgramDescription.invoke(** > >>> ProgramDriver.java:68) > >>> at org.apache.hadoop.util.**ProgramDriver.driver(** > >>> ProgramDriver.java:139) > >>> at > org.apache.mahout.driver.**MahoutDriver.main(**MahoutDriver.java:195) > >>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native Method) > >>> at > >>> sun.reflect.**NativeMethodAccessorImpl.**invoke(** > >>> NativeMethodAccessorImpl.java:**39) > >>> at > >>> sun.reflect.**DelegatingMethodAccessorImpl.**invoke(** > >>> DelegatingMethodAccessorImpl.**java:25) > >>> at java.lang.reflect.Method.**invoke(Method.java:597) > >>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:156) > >>> > >> > >> > >> > > >
