Hi Matthias, It possibly doesn't make sense to disable it, but then i'm sure someone will come up with a reason they don't want it! I'm happy to change it such that the checkpoint interval must be > 0.
Cheers, Damian On Fri, 3 Feb 2017 at 01:29 Matthias J. Sax <matth...@confluent.io> wrote: > Thanks Damian. > > One more question: "Checkpointing is disabled if the checkpoint interval > is set to a value <=0." > > > Does it make sense to disable check pointing? What's the tradeoff here? > > > -Matthias > > > On 2/2/17 1:51 AM, Damian Guy wrote: > > Hi Matthias, > > > > Thanks for the comments. > > > > 1. TBD - i need to do some performance tests and try and work out a > > sensible default. > > 2. Yes, you are correct. It could be a multiple of the > commit.interval.ms. > > But, that would also mean if you change the commit interval - say you > lower > > it, then you might also need to change the checkpoint setting (i.e, you > > still only want to checkpoint every n minutes). > > > > On Wed, 1 Feb 2017 at 23:46 Matthias J. Sax <matth...@confluent.io> > wrote: > > > >> Thanks for the KIP Damian. > >> > >> I am wondering about two things: > >> > >> 1. what should be the default value for the new parameter? > >> 2. why is the new parameter provided in ms? > >> > >> About (2): because > >> > >> "the minimum checkpoint interval will be the value of > >> commit.interval.ms. In effect the actual checkpoint interval will be a > >> multiple of the commit interval" > >> > >> it might be easier to just use an parameter that is "number-or-commit > >> intervals". > >> > >> > >> -Matthias > >> > >> > >> On 2/1/17 7:29 AM, Damian Guy wrote: > >>> Thanks for the comments Eno. > >>> As for exactly once, i don't believe this matters as we are just > >> restoring > >>> the change-log, i.e, the result of the aggregations that previously ran > >>> etc. So once initialized the state store will be in the same state as > it > >>> was before. > >>> Having the checkpoint in a kafka topic is not ideal as the state is per > >>> kafka streams instance. So each instance would need to start with a > >> unique > >>> id that is persistent. > >>> > >>> Cheers, > >>> Damian > >>> > >>> On Wed, 1 Feb 2017 at 13:20 Eno Thereska <eno.there...@gmail.com> > wrote: > >>> > >>>> As a follow up to my previous comment, have you thought about writing > >> the > >>>> checkpoint to a topic instead of a local file? That would have the > >>>> advantage that all metadata continues to be managed by Kafka, as well > as > >>>> fit with EoS. The potential disadvantage would be a slower latency, > >> however > >>>> if it is periodic as you mention, I'm not sure that would be a show > >> stopper. > >>>> > >>>> Thanks > >>>> Eno > >>>>> On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> > wrote: > >>>>> > >>>>> Thanks Damian, this is a good idea and will reduce the restore time. > >>>> Looking forward, with exactly once and support for transactions in > >> Kafka, I > >>>> believe we'll have to add some support for rolling back checkpoints, > >> e.g., > >>>> when a transaction is aborted. We need to be aware of that and ideally > >>>> anticipate a bit those needs in the KIP. > >>>>> > >>>>> Thanks > >>>>> Eno > >>>>> > >>>>> > >>>>>> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: > >>>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> I would like to start the discussion on KIP-116: > >>>>>> > >>>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration > >>>>>> > >>>>>> Thanks, > >>>>>> Damian > >>>>> > >>>> > >>>> > >>> > >> > >> > > > >
