Glad to hear about the progress. I am probably going to work on my algo soon and try to run it with very big graphs. I plan to do some profiling and experimental modifications to Giraph code as well to get it to scale. (Don't worry, I am not committing anything :)). Will keep you updated.
On Sep 27, 2011, at 4:33 PM, Jake Mannix wrote: > > > On Tue, Sep 27, 2011 at 1:29 PM, Aapo Kyrola <[email protected]> wrote: > > > The code is still in draft stage, but I attached it. > > Cool, thanks. The best place to usually attach it is to a JIRA ticket which > describes what > the code does, usually. > > it is actually quite optimized code with regards to data serialization etc. > > It is just that Giraph currently takes a lot of memory (I guess it is the > RPC), which makes it difficult > to run algos like this that cannot use a combiner. > > https://issues.apache.org/jira/browse/GIRAPH-28 > > is tracking some work toward reducing the memory footprint, but it requires > some work on > > https://issues.apache.org/jira/browse/GIRAPH-36 > > before it'll work as well as it can. Similarly, the RPC (including memory > overhead) is being improved in explorations in > > https://issues.apache.org/jira/browse/GIRAPH-12 > > and > > https://issues.apache.org/jira/browse/GIRAPH-37 > > Watch those spaces for upcoming improvements! > > -jake > > > Aapo > > On Sep 27, 2011, at 4:14 PM, Jake Mannix wrote: > >> >> >> On Tue, Sep 27, 2011 at 12:31 PM, Aapo Kyrola <[email protected]> wrote: >> >> I have written a very simple Belief Propagation algorithm for binary >> variables. >> >> Not ready for prime-time either, though :). >> >> That's awesome! Where's the code? :) >> >> It shouldn't be construed that "not ready for prime-time" is *bad*, in case >> that's what it looked like I was saying. >> >> More examples the better, so we can see where the bottlenecks are, and move >> them toward productionalized stage! >> >> -jake >> >> >> Aapo >> >> On Sep 27, 2011, at 3:03 PM, Jake Mannix wrote: >> >>> Not really. It's really really early, and they're in the "examples" stage >>> - nothing is >>> really productionized. There's things like PageRank, finding shortest >>> path, but >>> nothing is really ready for prime time yet. >>> >>> On Tue, Sep 27, 2011 at 11:56 AM, Josh Patterson <[email protected]> wrote: >>> Is there a list of known algorithms that have been implemented on the >>> Giraph framework? >>> >>> JP >>> >>> -- >>> Twitter: @jpatanooga >>> Solution Architect @ Cloudera >>> hadoop: http://www.cloudera.com >>> >> >> Aapo Kyrola >> Ph.D. student, http://www.cs.cmu.edu/~akyrola >> >> > > Aapo Kyrola > Ph.D. student, http://www.cs.cmu.edu/~akyrola > > > Aapo Kyrola Ph.D. student, http://www.cs.cmu.edu/~akyrola
