Re: Get after put in stateStore returns null

2020-04-04 Thread Jan Bols
Ok, Matthias, thanks for the clarification. This makes sense to me. Glad I learned something new about kafka-streams. Even if it was the hard way ;-) Greetings Jan On Wed, Apr 1, 2020 at 11:52 PM Matthias J. Sax wrote: > That is expected behavior. > > And yes, there is a `Transformer` instance

Re: Get after put in stateStore returns null

2020-04-01 Thread Sachin Mittal
You can try to check the file which rocksdb creates for the state store to see if value is stored for the key. What I suspect is that get may be getting called before put in your topology. Try retrieving the value from state store further downstream. Thanks Sachin On Thu, 2 Apr 2020, 02:35 Jan

Re: Get after put in stateStore returns null

2020-04-01 Thread Matthias J. Sax
That is expected behavior. And yes, there is a `Transformer` instance per partition with it's own store that holds one shard of the overall state. The reason is, that you could run one KafkaStreams instance per partition on different hosts/servers and thus, we need to have a `Transformer` and stat

Re: Get after put in stateStore returns null

2020-04-01 Thread Jan Bols
Ok, Matthias, thanks for the hint: *Even if any upstream operation was key-changing, no auto-repartition is triggered. If repartitioning is required, a call to through() should be performe

Re: Get after put in stateStore returns null

2020-03-26 Thread Matthias J. Sax
Your code looks correct to me. If you write into the store, you should also be able to read it back from the store. Can you reproduce the issue using `TopologyTestDriver`? How many partitions does your input topic have? Is your stream partitioned by key? Note that `transfrom()` does not do auto-re

Get after put in stateStore returns null

2020-03-25 Thread Jan Bols
Hi all, I'm trying to aggregate a stream of messages and return a stream of aggregated results using kafka streams. At some point, depending on the incoming message, the old aggregate needs to be closed and a new aggregate needs to be created, just like a session that is closed due to some close ev

Get after put in stateStore returns null

2020-03-24 Thread Jan Bols
Hi all, I'm trying to aggregate a stream of messages and return a stream of aggregated results using kafka streams. At some point, depending on the incoming message, the old aggregate needs to be closed and a new aggregate needs to be created, just like a session that is closed due to some close ev