[
https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415377#comment-13415377
]
Eli Reisman commented on GIRAPH-249:
------------------------------------
I just ran the latest 249 patch with 21% of the data I was able to run just
using trunk + patch 246, 256, and 250 this weekend. It died in the first couple
minutes on input super step with this listing on the failed workers:
Jul 16, 2012 4:48:52 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.HashMap.newKeyIterator(HashMap.java:840)
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:33)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:157)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Jul 16, 2012 4:49:14 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jul 16, 2012 4:48:59 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at java.lang.StringBuilder.toString(StringBuilder.java:430)
at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:232)
at sun.net.www.protocol.file.Handler.parseURL(Handler.java:50)
at java.net.URL.<init>(URL.java:596)
at java.net.URL.<init>(URL.java:464)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:977)
at sun.misc.URLClassPath.getResource(URLClassPath.java:169)
at java.net.URLClassLoader$1.run(URLClassLoader.java:194)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.jboss.netty.channel.AdaptiveReceiveBufferSizePredictorFactory.getPredictor(AdaptiveReceiveBufferSizePredictorFactory.java:65)
at
org.jboss.netty.channel.socket.nio.DefaultNioSocketChannelConfig.getReceiveBufferSizePredictor(DefaultNioSocketChannelConfig.java:148)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:305)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:274)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:194)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I will run a couple more tests with less data/smaller graph.splitmb to see
where the line is, and examine the Graphite metrics too.
> Move part of the graph out-of-core when memory is low
> -----------------------------------------------------
>
> Key: GIRAPH-249
> URL: https://issues.apache.org/jira/browse/GIRAPH-249
> Project: Giraph
> Issue Type: Improvement
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch,
> GIRAPH-249.patch, GIRAPH-249.patch
>
>
> There has been some talk about Giraph's scaling limitations due to keeping
> the whole graph and messages in RAM.
> We need to investigate methods to fall back to disk when running out of
> memory, while gracefully degrading performance.
> This issue is for graph storage. Messages should probably be a separate
> issue, although the interplay between the two is crucial.
> We should also discuss what are our primary goals here: completing a job
> (albeit slowly) instead of failing when the graph is too big, while still
> encouraging memory optimizations and high-memory clusters; or restructuring
> Giraph to be as efficient as possible in disk mode, making it almost a
> standard way of operating.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
