Hi Matthias, Thanks for the comments.
1. TBD - i need to do some performance tests and try and work out a sensible default. 2. Yes, you are correct. It could be a multiple of the commit.interval.ms. But, that would also mean if you change the commit interval - say you lower it, then you might also need to change the checkpoint setting (i.e, you still only want to checkpoint every n minutes). On Wed, 1 Feb 2017 at 23:46 Matthias J. Sax <matth...@confluent.io> wrote: > Thanks for the KIP Damian. > > I am wondering about two things: > > 1. what should be the default value for the new parameter? > 2. why is the new parameter provided in ms? > > About (2): because > > "the minimum checkpoint interval will be the value of > commit.interval.ms. In effect the actual checkpoint interval will be a > multiple of the commit interval" > > it might be easier to just use an parameter that is "number-or-commit > intervals". > > > -Matthias > > > On 2/1/17 7:29 AM, Damian Guy wrote: > > Thanks for the comments Eno. > > As for exactly once, i don't believe this matters as we are just > restoring > > the change-log, i.e, the result of the aggregations that previously ran > > etc. So once initialized the state store will be in the same state as it > > was before. > > Having the checkpoint in a kafka topic is not ideal as the state is per > > kafka streams instance. So each instance would need to start with a > unique > > id that is persistent. > > > > Cheers, > > Damian > > > > On Wed, 1 Feb 2017 at 13:20 Eno Thereska <eno.there...@gmail.com> wrote: > > > >> As a follow up to my previous comment, have you thought about writing > the > >> checkpoint to a topic instead of a local file? That would have the > >> advantage that all metadata continues to be managed by Kafka, as well as > >> fit with EoS. The potential disadvantage would be a slower latency, > however > >> if it is periodic as you mention, I'm not sure that would be a show > stopper. > >> > >> Thanks > >> Eno > >>> On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> wrote: > >>> > >>> Thanks Damian, this is a good idea and will reduce the restore time. > >> Looking forward, with exactly once and support for transactions in > Kafka, I > >> believe we'll have to add some support for rolling back checkpoints, > e.g., > >> when a transaction is aborted. We need to be aware of that and ideally > >> anticipate a bit those needs in the KIP. > >>> > >>> Thanks > >>> Eno > >>> > >>> > >>>> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: > >>>> > >>>> Hi all, > >>>> > >>>> I would like to start the discussion on KIP-116: > >>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration > >>>> > >>>> Thanks, > >>>> Damian > >>> > >> > >> > > > >
