Hello,
I am trying to run triangle count program in giraph using a hadoop cluster
of 6 machines each having 16 gb of ram. My data set is of size 1 gb which
is of json format.
The issue is I am getting OutOfMemoryError while trying to run this. For
data which is up to 25 thousand line of code it runs fine.I have updated
Hadoop-env.sh with export HADOOP_HEAPSIZE=2000 and mapred-site.xml with
<name>mapred.child.java.opts</name>
<value>-Xmx3000m -XX:+UseConcMarkSweepGC</value>
and the number of map instance is max 3 so for each machine will occupy
13 gb
But even after doing all this things I am getting the below exception,
Error from attempt_201404140400_0001_m_000011_0:
java.lang.IllegalStateException: run: Caught an unrecoverable exception
waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
at
org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111)
at
org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73)
at
org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192)
at
org.apache.giraph.graph.GraphTaskManager.processGraphPartitions(GraphTaskManager.java:753)
at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:273)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at
org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271)
at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space
Thanks,
Arun
