How about Scribing messages (and writing to HDFS) during calculation? Then you could perform bulk log analysis on the output with a separate Hadoop (or Pig) job.
http://en.wikipedia.org/wiki/Scribe_(log_server) Andy On Thu, Sep 22, 2011 at 7:31 AM, Claudio Martella <claudio.marte...@gmail.com> wrote: > Hi Avery, > > thanks, yes it does. The question would be though how to share the > file handle between the vertices on the same node. i could open the > file on the preApplication() and close it on the postApplication() but > i would end up potentially with as many files as vertices in the > graph. > > Do you have any idea on this side? Maybe share somehow the handle and a lock? > > On Thu, Sep 22, 2011 at 4:07 PM, Avery Ching <ach...@apache.org> wrote: >> There are some methods in Vertex (i.e. preApplication(), preSuperstep(), >> postApplication(), postSuperstep()) that can be overidden to do anything you >> like, for instance write out some data to an HDFS file. We have an open >> issue on outputting Aggregator values that is unassigned if you'd like to >> take a look at it as well (https://issues.apache.org/jira/browse/GIRAPH-10). >> >> Hope this helps, >> >> Avery >> >> On 9/22/11 7:34 AM, Claudio Martella wrote: >>> >>> Hello list, >>> >>> I have the need to emit to HDFS once in a while some Text. This >>> doesn't happen necessarily at the end of the computation and I might >>> need to emit something more complex than just the VertexValue, so I'd >>> like more control than what the VertexWriter gives me. >>> >>> What do you suggest I might do to obtain a handler to a HDFS file (it >>> can be in parts aswell) to write to? >>> Is there any code I can start looking at? >>> >>> Thanks! >>> Claudio >>> >> >> > > > > -- > Claudio Martella > claudio.marte...@gmail.com >
