Hi
while executing triangle count progrm over graph in a cluster of 8 server
machines i am getting out of memory exception.My data size is of 1GB
In my conf/hadoop-env.sh file i have updated the heapsize
export HADOOP_HEAPSIZE=6000
even after that i am getting the below exception
Can someone help ?
at org\.apache\.giraph\.graph\.GraphMapper\.run(GraphMapper\.java:102)
at org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:764)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:364)
at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
at java\.security\.AccessController\.doPrivileged(Native Method)
at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
at
org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1190)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
Caused by: java\.lang\.IllegalStateException: waitFor: ExecutionException
occurred while waiting for
org\.apache\.giraph\.utils\.ProgressableUtils$FutureWaitable@2a0850f8
at
org\.apache\.giraph\.utils\.ProgressableUtils\.waitFor(ProgressableUtils\.java:151)
at
org\.apache\.giraph\.utils\.ProgressableUtils\.waitForever(ProgressableUtils\.java:111)
at
org\.apache\.giraph\.utils\.ProgressableUtils\.getFutureResult(ProgressableUtils\.java:73)
at
org\.apache\.giraph\.utils\.ProgressableUtils\.getResultsWithNCallables(ProgressableUtils\.java:192)
at
org\.apache\.giraph\.graph\.GraphTaskManager\.processGraphPartitions(GraphTaskManager\.java:753)
at
org\.apache\.giraph\.graph\.GraphTaskManager\.execute(GraphTaskManager\.java:273)
at org\.apache\.giraph\.graph\.GraphMapper\.run(GraphMapper\.java:92)
\.\.\. 7 more
Caused by: java\.util\.concurrent\.ExecutionException:
java\.lang\.OutOfMemoryError: Java heap space
at
java\.util\.concurrent\.FutureTask$Sync\.innerGet(FutureTask\.java:232)
at java\.util\.concurrent\.FutureTask\.get(FutureTask\.java:91)
at
org\.apache\.giraph\.utils\.ProgressableUtils$FutureWaitable\.waitFor(ProgressableUtils\.java:271)
at
org\.apache\.giraph\.utils\.ProgressableUtils\.waitFor(ProgressableUtils\.java:143)
\.\.\. 13 more
Caused by: java\.lang\.OutOfMemoryError: Java heap space
at
org\.apache\.giraph\.utils\.UnsafeByteArrayOutputStream\.ensureSize(UnsafeByteArrayOutputStream\.java:131)
at
org\.apache\.giraph\.utils\.UnsafeByteArrayOutputStream\.writeLong(UnsafeByteArrayOutputStream\.java:234)
at org\.apache\.hadoop\.io\.LongWritable\.write(LongWritable\.java:42)
at
org\.apache\.giraph\.comm\.messages\.ByteArrayMessagesPerVertexStore\.addPartitionMessages(ByteArrayMessagesPerVertexStore\.java:121)
at
org\.apache\.giraph\.comm\.requests\.SendWorkerMessagesRequest\.doRequest(SendWorkerMessagesRequest\.java:72)
at
org\.apache\.giraph\.comm\.netty\.NettyWorkerClientRequestProcessor\.doRequest(NettyWorkerClientRequestProcessor\.java:470)
at
org\.apache\.giraph\.comm\.netty\.NettyWorkerClientRequestProcessor\.sendMessageRequest(NettyWorkerClientRequestProcessor\.java:186)
at org\.apache\.giraph\.graph\.Vertex\.sendMessage(Vertex\.java:340)
at
org\.apache\.giraph\.graph\.Vertex\.sendMessageToAllEdges(Vertex\.java:353)
at
org\.apache\.giraph\.examples\.TriangleCounting\.compute(TriangleCounting\.java:47)
at
org\.apache\.giraph\.graph\.ComputeCallable\.computePartition(ComputeCallable\.java:234)
at
org\.apache\.giraph\.graph\.ComputeCallable\.call(ComputeCallable\.java:162)
at
org\.apache\.giraph\.graph\.ComputeCallable\.call(ComputeCallable\.java:70)
at
org\.apache\.giraph\.utils\.LogStacktraceCallable\.call(LogStacktraceCallable\.java:51)
at
java\.util\.concurrent\.FutureTask$Sync\.innerRun(FutureTask\.java:303)
at java\.util\.concurrent\.FutureTask\.run(FutureTask\.java:138)
at
java\.util\.concurrent\.ThreadPoolExecutor$Worker\.runTask(ThreadPoolExecutor\.java:895)
at
java\.util\.concurrent\.ThreadPoolExecutor$Worker\.run(ThreadPoolExecutor\.java:918)
at java\.lang\.Thread\.run(Thread\.java:662)
,Task attempt_201404040451_0001_m_000001_0 failed to report status for 600
seconds\. Killing!" .
