Thanks for the feedback. As a matter of fact that's exactly the type of functionality i'm looking for, with minimal infrastructure cost though. Thanks!
On Fri, Sep 23, 2011 at 7:13 PM, Andy Schlaikjer <ha...@twitter.com> wrote: > How about Scribing messages (and writing to HDFS) during calculation? > Then you could perform bulk log analysis on the output with a separate > Hadoop (or Pig) job. > > http://en.wikipedia.org/wiki/Scribe_(log_server) > > Andy > > > On Thu, Sep 22, 2011 at 7:31 AM, Claudio Martella > <claudio.marte...@gmail.com> wrote: >> Hi Avery, >> >> thanks, yes it does. The question would be though how to share the >> file handle between the vertices on the same node. i could open the >> file on the preApplication() and close it on the postApplication() but >> i would end up potentially with as many files as vertices in the >> graph. >> >> Do you have any idea on this side? Maybe share somehow the handle and a lock? >> >> On Thu, Sep 22, 2011 at 4:07 PM, Avery Ching <ach...@apache.org> wrote: >>> There are some methods in Vertex (i.e. preApplication(), preSuperstep(), >>> postApplication(), postSuperstep()) that can be overidden to do anything you >>> like, for instance write out some data to an HDFS file. We have an open >>> issue on outputting Aggregator values that is unassigned if you'd like to >>> take a look at it as well (https://issues.apache.org/jira/browse/GIRAPH-10). >>> >>> Hope this helps, >>> >>> Avery >>> >>> On 9/22/11 7:34 AM, Claudio Martella wrote: >>>> >>>> Hello list, >>>> >>>> I have the need to emit to HDFS once in a while some Text. This >>>> doesn't happen necessarily at the end of the computation and I might >>>> need to emit something more complex than just the VertexValue, so I'd >>>> like more control than what the VertexWriter gives me. >>>> >>>> What do you suggest I might do to obtain a handler to a HDFS file (it >>>> can be in parts aswell) to write to? >>>> Is there any code I can start looking at? >>>> >>>> Thanks! >>>> Claudio >>>> >>> >>> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> > -- Claudio Martella claudio.marte...@gmail.com
