Re: kafka streams checkpoint and restore understanding

2017-06-10 Thread Matthias J. Sax
Yep. That's correct. Cool that you are writing a book! :) -Matthias On 6/9/17 7:07 PM, john cheng wrote: > Tks Matthias, I add some offset log in ProcessorStateManager and > RocksDBStore. > The behaviour is just like you explaination. > > This is restart instance1's log. The two active task re

Re: kafka streams checkpoint and restore understanding

2017-06-09 Thread john cheng
Tks Matthias, I add some offset log in ProcessorStateManager and RocksDBStore. The behaviour is just like you explaination. This is restart instance1's log. The two active task really doing replay work from position 2 of checkpoint file [06:55,061] createStreamTask 0_0, partitions: [streams-wc-in

Re: kafka streams checkpoint and restore understanding

2017-06-09 Thread Matthias J. Sax
Your observation is completely correct and this is also correct behavior. Note, that instance1 and instance2 both also do have a local RocksDB instance that holds the state. The checkpoint file basically tells streams, what prefix of the changelog topic is already in RocksDB. As Streams loads (no

kafka streams checkpoint and restore understanding

2017-06-07 Thread john cheng
I have two app instance, input topic has 2 partitions, each instance config one thread and one replicas. also, instance1's state-store is /tmp/kafka-streams, instance2's state-store is /tmp/kafka-streams2. now I do this experiment to study checkpointin kafka streams (0.10.0.0). 1. start instance1,