Thanks Damian. One more question: "Checkpointing is disabled if the checkpoint interval is set to a value <=0."
Does it make sense to disable check pointing? What's the tradeoff here? -Matthias On 2/2/17 1:51 AM, Damian Guy wrote: > Hi Matthias, > > Thanks for the comments. > > 1. TBD - i need to do some performance tests and try and work out a > sensible default. > 2. Yes, you are correct. It could be a multiple of the commit.interval.ms. > But, that would also mean if you change the commit interval - say you lower > it, then you might also need to change the checkpoint setting (i.e, you > still only want to checkpoint every n minutes). > > On Wed, 1 Feb 2017 at 23:46 Matthias J. Sax <matth...@confluent.io> wrote: > >> Thanks for the KIP Damian. >> >> I am wondering about two things: >> >> 1. what should be the default value for the new parameter? >> 2. why is the new parameter provided in ms? >> >> About (2): because >> >> "the minimum checkpoint interval will be the value of >> commit.interval.ms. In effect the actual checkpoint interval will be a >> multiple of the commit interval" >> >> it might be easier to just use an parameter that is "number-or-commit >> intervals". >> >> >> -Matthias >> >> >> On 2/1/17 7:29 AM, Damian Guy wrote: >>> Thanks for the comments Eno. >>> As for exactly once, i don't believe this matters as we are just >> restoring >>> the change-log, i.e, the result of the aggregations that previously ran >>> etc. So once initialized the state store will be in the same state as it >>> was before. >>> Having the checkpoint in a kafka topic is not ideal as the state is per >>> kafka streams instance. So each instance would need to start with a >> unique >>> id that is persistent. >>> >>> Cheers, >>> Damian >>> >>> On Wed, 1 Feb 2017 at 13:20 Eno Thereska <eno.there...@gmail.com> wrote: >>> >>>> As a follow up to my previous comment, have you thought about writing >> the >>>> checkpoint to a topic instead of a local file? That would have the >>>> advantage that all metadata continues to be managed by Kafka, as well as >>>> fit with EoS. The potential disadvantage would be a slower latency, >> however >>>> if it is periodic as you mention, I'm not sure that would be a show >> stopper. >>>> >>>> Thanks >>>> Eno >>>>> On 1 Feb 2017, at 12:58, Eno Thereska <eno.there...@gmail.com> wrote: >>>>> >>>>> Thanks Damian, this is a good idea and will reduce the restore time. >>>> Looking forward, with exactly once and support for transactions in >> Kafka, I >>>> believe we'll have to add some support for rolling back checkpoints, >> e.g., >>>> when a transaction is aborted. We need to be aware of that and ideally >>>> anticipate a bit those needs in the KIP. >>>>> >>>>> Thanks >>>>> Eno >>>>> >>>>> >>>>>> On 1 Feb 2017, at 10:18, Damian Guy <damian....@gmail.com> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I would like to start the discussion on KIP-116: >>>>>> >>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-116+-+Add+State+Store+Checkpoint+Interval+Configuration >>>>>> >>>>>> Thanks, >>>>>> Damian >>>>> >>>> >>>> >>> >> >> >
signature.asc
Description: OpenPGP digital signature
