Thanks very much, so are you saying if I use Dgiraph.maxPartitionsInMemory and Dgiraph.maxMessagesInMemory to make them both smaller number, then it might work?
Thanks again, Jian On Thu, Oct 17, 2013 at 12:56 AM, Jyotirmoy Sundi <[email protected]>wrote: > You need to tune it per your cluster. This is what mentioned in the docs: > *"It is difficult to decide a general policy to use out-of-core > capabilities*, as it depends on the behavior of the algorithm and the > input graph. The exact number of partitions and messages to keep in memory > depends on the cluster capabilities, the number of messages produced per > superstep, and number of active vertices per superstep. Moreover, it > depends on the type and size of vertex values and messages. For example, > algorithms such as Belief Propagation tend to keep large vertex values, > while algorithms such as clique computations tend to send large messages > along. Hence, it depends on your algorithm what feature to rely on more." > > Thanks > Sundi > > > On Wed, Oct 16, 2013 at 9:41 PM, Jianqiang Ou <[email protected]>wrote: > >> Hi Sundi, >> >> I just tried your method, but somehow the job failed, the attached is the >> history of the job. and it was good without the outofcore options. Do you >> have any clue why is that? >> >> The command I used to run the program is below: >> >> $HADOOP_HOME/bin/hadoop jar >> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner -Dgiraph.useOutOfCoreMessages=true >> -Dgiraph.useOutOfCoreGraph=true >> org.apache.giraph.examples.SimplePageRankComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> -vip /user/andy/input/tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /user/andy/output/page3 -w 3 -mc >> org.apache.giraph.examples.SimplePageRankComputation\$SimplePageRankMasterCompute >> >> Many thanks, >> >> Jianqiang >> >> On Wed, Oct 16, 2013 at 12:11 PM, Jianqiang Ou >> <[email protected]>wrote: >> >>> got it, thank you very much! >>> >>> >>> On Wed, Oct 16, 2013 at 10:43 AM, Jyotirmoy Sundi <[email protected]>wrote: >>> >>>> Put it as -Dgiraph.useOutOfCoreMessages=true >>>> -Dgiraph.useOutOfCoreGraph=true after GiraphRuuner >>>> like >>>> hadoop jar girap.jar org.apache.giraph.GiraphRunner >>>> -Dgiraph.useOutOfCoreMessages=true >>>> -Dgiraph.useOutOfCoreGraph=true ... >>>> >>>> >>>> >>>> >>>> On Wed, Oct 16, 2013 at 7:29 AM, Jianqiang Ou <[email protected] >>>> > wrote: >>>> >>>>> Hi I have a question about the out of core giraph. It is said that, in >>>>> order to use disk to store the partions, we need to use " >>>>> giraph.useOutOfCoreGraph=true", but where should I put this statement >>>>> to? >>>>> >>>>> BTW, I am just trying to use the pagerank or shortestpath example to >>>>> test the out of core performance of my cluster. >>>>> >>>>> Thanks very much, >>>>> Jian >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Jyotirmoy Sundi >>>> Data Engineer, >>>> Admobius >>>> >>>> San Francisco, CA 94158 >>>> >>> >>> >> On Wed, Oct 16, 2013 at 12:11 PM, Jianqiang Ou >> <[email protected]>wrote: >> >>> got it, thank you very much! >>> >>> >>> On Wed, Oct 16, 2013 at 10:43 AM, Jyotirmoy Sundi <[email protected]>wrote: >>> >>>> Put it as -Dgiraph.useOutOfCoreMessages=true >>>> -Dgiraph.useOutOfCoreGraph=true after GiraphRuuner >>>> like >>>> hadoop jar girap.jar org.apache.giraph.GiraphRunner >>>> -Dgiraph.useOutOfCoreMessages=true >>>> -Dgiraph.useOutOfCoreGraph=true ... >>>> >>>> >>>> >>>> >>>> On Wed, Oct 16, 2013 at 7:29 AM, Jianqiang Ou <[email protected] >>>> > wrote: >>>> >>>>> Hi I have a question about the out of core giraph. It is said that, in >>>>> order to use disk to store the partions, we need to use " >>>>> giraph.useOutOfCoreGraph=true", but where should I put this statement >>>>> to? >>>>> >>>>> BTW, I am just trying to use the pagerank or shortestpath example to >>>>> test the out of core performance of my cluster. >>>>> >>>>> Thanks very much, >>>>> Jian >>>>> >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Jyotirmoy Sundi >>>> Data Engineer, >>>> Admobius >>>> >>>> San Francisco, CA 94158 >>>> >>> >>> >> > > > -- > Best Regards, > Jyotirmoy Sundi > Data Engineer, > Admobius > > San Francisco, CA 94158 >
