Re: How to properly use a clean a TimestampedKeyValueStore

2022-01-10 Thread Guozhang Wang
Hi Miguel, I checked your code and it seems fine to me, so I would not suspect anything with your logic. The next thing I'd suggest to check if you have many cases where the same key gets deleted and then re-inserted (you can add some logging at the `put` and `delete` calls`). Guozhang On Tue,

Re: How to properly use a clean a TimestampedKeyValueStore

2022-01-04 Thread Miguel González
Hello Here's my store definition and my transformer with the whole logic (I included the transformer with issues and another version (V2) of the transformer with some improvements that I believe are necessary) https://gist.github.com/magg/576bf3381c9c0501b9761b54e9d86375 Thanks - Miguel On

Re: How to properly use a clean a TimestampedKeyValueStore

2022-01-04 Thread Guozhang Wang
Hi Miguel, How is your kvStore being constructed? Could you paste the snippet of the related construction code, as well as the related iterating / deletion code here? On Tue, Jan 4, 2022 at 2:25 PM Matthias J. Sax wrote: > Not 100% sure. From what you describe it should work as expected. > >

Re: How to properly use a clean a TimestampedKeyValueStore

2022-01-04 Thread Matthias J. Sax
Not 100% sure. From what you describe it should work as expected. It seems `delete()` does not delete the key from the store (ie, RocksDB) itself (for unknown reasons)? Are you closing all your iterators correctly? (More or less a wild guess at the moment.) Did you enable caching for the

How to properly use a clean a TimestampedKeyValueStore

2021-12-24 Thread Miguel González
Hello I'm using Kafka Streams and I have a transformer that uses a TimestampedKeyValueStore, I have a punctuator that is in charge of cleaning the store, Basically I'm iterating the store using kvStore.all() and deleting the keys based on some logic with kvStore.delete(key); I'm seeing the