Claudio,

For the AggregatorWriter, shouldn't your case be addressed? Since only a single thread will use the AggregatorWriter, it can dump the aggregators to HDFS at every superstep. Am I missing something?


Avery

On 9/30/11 2:20 AM, Claudio Martella wrote:
The first option sounds like a good idea to me. It fits quite directly
with the current interface.

This design though doesn't fit with my current need for dumping data
to disk. I'll write an email about it with a couple of new questions.

I think that with some help I can produce some code on this JIRA.

On Thu, Sep 29, 2011 at 9:49 PM, Avery Ching (Commented) (JIRA)
<j...@apache.org>  wrote:
    [ 
https://issues.apache.org/jira/browse/GIRAPH-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117569#comment-13117569
 ]

Avery Ching commented on GIRAPH-10:
-----------------------------------

This is definitely an open question here.  Here's a quick thought off the top 
of my head.

How about something like:

public interface AggregatorsWriter {
    void initialize(TaskAttemptContext context, int superstep) throws 
IOException;

    void writeAggregators(Collection<Aggregator<? extends Writable>>  
aggregators)
        throws IOException, InterruptedException;

    void close(TaskAttemptContext context)
        throws IOException, InterruptedException;
}

Then we could define a method that lets the user select the frequency of how 
frequently to write the aggregators maybe with an enum (ALL_SUPERSTEPS, 
LAST_SUPERSTEP...).

We could easily implement a default AggregatorsWriter that simply dumps the 
name and aggregator values with the registered aggregator name (maybe class 
name too) and then the value.toString().  And users can implement something 
better if they like.

Another alternative would be to support writing aggregators separately by the 
type, but might be a little excessive and could be done later....hopefully 
someone else also chimes in with some thoughts.

Aggregators are not exported
----------------------------

                 Key: GIRAPH-10
                 URL: https://issues.apache.org/jira/browse/GIRAPH-10
             Project: Giraph
          Issue Type: New Feature
            Reporter: Avery Ching
            Priority: Minor

Currently, aggregator values cannot be saved after a Giraph job.  There should 
be a way to do this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira






Reply via email to