Hi Claudio,
I think I understand what you are trying to do, a kind of a distributed
logging for debugging. I think such a feature can definitely be
useful. Aggregators might be able to do what you want, then with things
like https://issues.apache.org/jira/browse/GIRAPH-10, perhaps not just
Thanks for the feedback. As a matter of fact that's exactly the type
of functionality i'm looking for, with minimal infrastructure cost
though. Thanks!
On Fri, Sep 23, 2011 at 7:13 PM, Andy Schlaikjer wrote:
> How about Scribing messages (and writing to HDFS) during calculation?
> Then you could
i'm really just trying to emit "results" into an hdfs file at
different moments of the computation. I'm really just thinking at a
functionality like log.debug(), to give an example, where all the
messages are collected from different workers at different supersteps.
At the moment I've implemented t
How about Scribing messages (and writing to HDFS) during calculation?
Then you could perform bulk log analysis on the output with a separate
Hadoop (or Pig) job.
http://en.wikipedia.org/wiki/Scribe_(log_server)
Andy
On Thu, Sep 22, 2011 at 7:31 AM, Claudio Martella
wrote:
> Hi Avery,
>
> thank
Hi Avery,
thanks, yes it does. The question would be though how to share the
file handle between the vertices on the same node. i could open the
file on the preApplication() and close it on the postApplication() but
i would end up potentially with as many files as vertices in the
graph.
Do you ha
There are some methods in Vertex (i.e. preApplication(), preSuperstep(),
postApplication(), postSuperstep()) that can be overidden to do anything
you like, for instance write out some data to an HDFS file. We have an
open issue on outputting Aggregator values that is unassigned if you'd
like t
Hello list,
I have the need to emit to HDFS once in a while some Text. This
doesn't happen necessarily at the end of the computation and I might
need to emit something more complex than just the VertexValue, so I'd
like more control than what the VertexWriter gives me.
What do you suggest I might
