Hi All, I looked at the RocksDB KV store implementation and I found that deserialization has to happen for each key lookup. Given a scenario where the key lookup has to happen for every single message would it still be a good idea to store it in rocksdb store or would in-memory store/cache be more efficient? I know if the data is stored in KV store it will automatically distribute when scale up/scale down happens and its fault tolerant.
For example, if there are 1M user events and a user config of size 1KB is persisted into rocksdb then for each event would the state have to be deserialized? Wouldn't this create so many garbage? Also, is there is per machine sort of state store which can be used for all keys that are sent to that task manager? Thanks
