GitHub user PragmaTwice added a comment to the discussion: TimeSeries Proposal
Some comments: - In Kvrocks there are several existing column families. And the most relevant CFs here are `metadata` (metadata CF) and `default` (subkey CF), where all keys inside should follow a special key format (user keys and sub keys, per https://kvrocks.apache.org/community/data-structure-on-rocksdb#user-key-encoding). I think it is clear that "Datameta" in the proposal is stored in `metadata` CF, but for other parts we can specify clearly whether it is stored in subkey CF or a new timeseries CF. - There may be some pros and cons for introducing a new CF (e.g. we face some [compatibility issue of introducing one](https://github.com/apache/kvrocks/discussions/2851)). If we introduce a CF for timeseries, we should consider: - How to avoid any key conflicts? e.g. in the proposal, there may be some conflicts since we didn't introduce some enum values as prefixes in the key format (for example, a key in "Reverse Index" format may be same as another key in "Label Index"). Also, we cannot iterate over a special type of keys if we don't distingush them by prefixing keys with some enum values. (Maybe you can find some idea in the encoding of search data: https://kvrocks.apache.org/community/kvrocks-search-index-encoding.) - How to do garbage-collection? We have key expiration and also timeseries-specific "retention", and we need to modify compaction filter (https://github.com/apache/kvrocks/blob/unstable/src/storage/compact_filter.cc) and maybe also compaction checker (https://github.com/apache/kvrocks/blob/unstable/src/storage/compaction_checker.cc) to (1): figure out which key is expired if a compaction is triggered, and (2): when we need to trigger a compaction (maybe there's timeseries specific logic). - Also found some reference to codebase of redis timeseries. We can of course read its code, but we cannot use them directly due to ASF license requirements. GitHub link: https://github.com/apache/kvrocks/discussions/3044#discussioncomment-13659303 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
