Thanks for your input.
The mapred-site.xml configuration as the following:
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx2G</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5G</value>
</property>
<property>
<name>mapred.cluster.map.memory.mb</name>
<value>7000</value>
</property>
<property>
<name>mapred.cluster.max.map.memory.mb</name>
<value>10000</value>
</property>
The exception occurs :(
I run the pagerank benchmark on my input i.e. 1m vertices as well as
shortest paths both works well before modifying the configurations ...
On 7/5/12, Avery Ching (JIRA) <[email protected]> wrote:
>
> [
> https://issues.apache.org/jira/browse/GIRAPH-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406882#comment-13406882
> ]
>
> Avery Ching commented on GIRAPH-234:
> ------------------------------------
>
> Jan's suggestion can also be used at runtime i.e.
> -Dmapred.child.java.opts=-Xmx4000m . I usually set -Xms as well to ensure I
> get the memory.
>
>> Error: GC overhead limit exceeded. Child Error
>> org.apache.hadoop.mapred.TaskRunner.run
>> --------------------------------------------------------------------------------------
>>
>> Key: GIRAPH-234
>> URL: https://issues.apache.org/jira/browse/GIRAPH-234
>> Project: Giraph
>> Issue Type: Bug
>> Components: bsp, examples, mapreduce
>> Affects Versions: 0.1.0
>> Reporter: Amani
>> Labels: example, giraph
>>
>> Hello everyone,
>> I'm trying to run pregel style implementation of strongly connected
>> components algorithm on top of giraph. A user defined writable
>> (LongPairWritable) used as the type of the messages between the vertices.
>> The algorithm works well in single node. However, whenever I tested using
>> large graph (around 1 million vertices) in cluster, a GC overhead limit
>> exceeded caused by out of memory exception. The error comes after
>> attempt=0 super-step=1 with 2 minutes pending. I'm wondering how to solve
>> the problem and what causes it?
>> Is it possible that LongPairWritable cause the problem?
>> I tried a simple test:
>> Succeeded job:
>> Comput(Iterator<LongPairWritable> msgIterator){
>> If(getSuperstep() >=20){
>> voteToHalt();
>> } else {
>> //Do nothing
>> }
>> }
>> Failed job:
>> Comput(Iterator<LongPairWritable> msgIterator){
>> If(getSuperstep() >=20){
>> voteToHalt();
>> } else {
>> sendToAll(new LongPairWritable(new LongWritable(0),new LongWritable(0));
>> }
>> }
>> Thanks in advance.
>> Amani
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
--
Amani AlOnazi
MSc Computer Science
King Abdullah University of Science and Technology
Kingdom of Saudi Arabia
[email protected] | +966 (0) 555 191 795
--
------------------------------
This message and its contents, including attachments are intended solely
for the original recipient. If you are not the intended recipient or have
received this message in error, please notify me immediately and delete
this message from your computer system. Any unauthorized use or
distribution is prohibited. Please consider the environment before printing
this email.
