[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692424#comment-16692424
]
Adam Bellemare commented on KAFKA-4212:
---------------------------------------
We have some business requirements where we need to store a large amount of
data in a KTable for interactive querying. After a period of time, we want to
time it out via TTL.
Currently we have an implementation of 2.0 with RocksDB using TTL. In addition,
we prevent records from loading from the changelog if recordTime + TTL <
referenceTimeStamp (default = System.currentTimeInMillis() ). We do not set any
retention properties besides the default on the changelog (such that if we
needed to change the retention period, we would not have to recompute
everything).
I believe that the ask of this ticket and the solution my company is using
intersect. As far as I can see, there is no KIP to address this specific issue.
Is there anything obvious preventing my solution from being a good candidate? I
will create a KIP and submit a PR if there are no obvious blockers, if this
ticket isn't out of date or overtaken by events.
> Add a key-value store that is a TTL persistent cache
> ----------------------------------------------------
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.10.0.1
> Reporter: Elias Levy
> Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
