Hi all, I am struggling with the settings to use out of core messages. I have 3 nodes with 16 GB RAM each ( one master, two workers).
I ran into a java heap space OOM Error. First question is: Where do I set the mapred.child.java-opts Options? Do I need to add them via the "-ca mapred.child….“ option or by using „-Dmapred.child…..“ I tried both, but nothing seems to work out. I run it on a cloudera cluster, and when looking in the web frontend I see, that it only uses 3 GB of my 16 GB RAM. Are those even the right options ? giraph.maxMessagesInMemory - is it per worker? Or what exactly is counted here? and how does it correlate to giraph.messagesBufferSize? I am really lost right now. My graph has currently only 8000 nodes and 70000 edges. During one step I need to send more than 15 000 000 messages and this is when I get the OOM error. I turned on the out of core messages feature without changing the above mentioned options and my computation really slowed down. I guess because it was writing 14 000 000 messages to disk Hope you can help me. Regards Pascal
