It would be good to find out why it runs out memory. It really
shouldn't take huge gobs of memory to run a LocalRunner and local
file-based word count over a few lines of text. I wonder if perhaps
something like io.sort.mb causes big buffer pre-allocations that could
be reduced for these tests with a simple setting.
Doug
On 06/24/2010 10:20 AM, Ken Krugler wrote:
I was running into an out-of-memory error while running the
org.apache.avro.mapred.TestWordCountSpecific and TestWordCountGeneric
tests:
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
[junit] at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
[junit] at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
[junit] at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
I added a maxmemory attribute to the junit task in the Java build.xml,
and it seemed to fix the problem:
<macrodef name="test-runner">
<attribute name="files.location" />
<attribute name="tests.pattern" />
<attribute name="test.dir" default="${test.java.build.dir}" />
<sequential>
<junit showoutput="yes"
printsummary="withOutAndErr"
haltonfailure="no"
fork="yes"
maxmemory="256m"
I can file a Jira issue and a patch, but seems pretty trivial...
-- Ken
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
