Re: More control over checkpoints

2015-02-06 Thread Chris Riccomini
nk you again, Best regards,Tom From: Chris Riccomini < > criccom...@apache.org> > To: Chris Riccomini > Cc: dev@samza.apache.org; Thomas Bernhardt > Sent: Friday, February 6, 2015 11:47 AM > Subject: Re: More control over checkpoints > > Hey Tom, > > > I&

Re: More control over checkpoints

2015-02-06 Thread Thomas Bernhardt
you again, Best regards,Tom From: Chris Riccomini To: Chris Riccomini Cc: dev@samza.apache.org; Thomas Bernhardt Sent: Friday, February 6, 2015 11:47 AM Subject: Re: More control over checkpoints Hey Tom, > I'm planning to have my external state also keep the topic+partition o

Re: More control over checkpoints

2015-02-06 Thread Chris Riccomini
Hey Tom, > I'm planning to have my external state also keep the topic+partition offsets in the same place that holds the aggregation state. Also, for this, you'll have to keep track of the SSP->offset mapping in your task. This can be done with a Map(). Every time an IncomingMessageEnvelope comes

Re: More control over checkpoints

2015-02-06 Thread Chris Riccomini
Hey Tom, > questions:Can my StreamTask receive a call when a checkpoint takes place so that it can write its state? The pattern for this is to take control of the checkpointing in your task, rather than using task.commit.ms. If you: # Set task.commit.ms=-1 # Set task.window.ms=6 # Implement