Hello Here's my store definition and my transformer with the whole logic (I included the transformer with issues and another version (V2) of the transformer with some improvements that I believe are necessary)
https://gist.github.com/magg/576bf3381c9c0501b9761b54e9d86375 Thanks - Miguel On Tue, Jan 4, 2022 at 5:21 PM Guozhang Wang <wangg...@gmail.com> wrote: > Hi Miguel, > > How is your kvStore being constructed? Could you paste the snippet of the > related construction code, as well as the related iterating / deletion code > here? > > On Tue, Jan 4, 2022 at 2:25 PM Matthias J. Sax <mj...@apache.org> wrote: > > > Not 100% sure. From what you describe it should work as expected. > > > > It seems `delete()` does not delete the key from the store (ie, RocksDB) > > itself (for unknown reasons)? > > > > Are you closing all your iterators correctly? (More or less a wild guess > > at the moment.) > > > > Did you enable caching for the store? (Just to double check if it could > > be caching related or not.) > > > > > > -Matthias > > > > > > On 12/24/21 11:08 AM, Miguel González wrote: > > > Hello > > > > > > I'm using Kafka Streams and I have a transformer that uses > > > a TimestampedKeyValueStore, I have a punctuator that is in charge of > > > cleaning the store, > > > > > > Basically I'm iterating the store using kvStore.all() and deleting the > > keys > > > based on some logic with kvStore.delete(key); > > > > > > I'm seeing the changelog topic for the store grow unbounded, I'm seeing > > > many values with null for the same keys... I think those are called > > > tombstones right? but the punctuator is constantly doing the same > thing > > > trying to delete the same keys.. I see more tombstones being inserted. > > > > > > Is this the expected behavior? If so, how can I correctly clean that > > store? > > > > > > thanks > > > - Miguel > > > > > > > > -- > -- Guozhang >
