[ https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16876516#comment-16876516 ]
Sophie Blee-Goldman commented on KAFKA-4212: -------------------------------------------- Hi [~jamesritt], I think your PR is a good first step towards a TTL KV store but it's not sufficient to simply expose the rocksdb TTL option. The changelog topic is the ultimate source of truth for a store, so if you don't somehow clean up old records there you will just restore all the old and deleted records on restore/rebalance. One solution might be to somehow store/retrieve the current time such that after a restore/rebalance you only load "valid" records (see KAFKA-7934) (also as a note, Rocks TTL is not "strict" in that is provides a lower, not upper, bound on how long data will remain in the db) > Add a key-value store that is a TTL persistent cache > ---------------------------------------------------- > > Key: KAFKA-4212 > URL: https://issues.apache.org/jira/browse/KAFKA-4212 > Project: Kafka > Issue Type: Improvement > Components: streams > Affects Versions: 0.10.0.1 > Reporter: Elias Levy > Priority: Major > Labels: api > > Some jobs needs to maintain as state a large set of key-values for some > period of time. I.e. they need to maintain a TTL cache of values potentially > larger than memory. > Currently Kafka Streams provides non-windowed and windowed key-value stores. > Neither is an exact fit to this use case. > The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as > required, but does not support expiration. The TTL option of RocksDB is > explicitly not used. > The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment > dropping, but it stores multiple items per key, based on their timestamp. > But this store can be repurposed as a cache by fetching the items in reverse > chronological order and returning the first item found. > KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here > we desire a variable-capacity memory-overflowing TTL caching store. > Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be > useful to have an official and proper TTL cache API and implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)