Hi Chirag That's an interesting idea. IIUC, storing key-values can be simply implemented for Redis, but supporting checkpoint and recovery is relatively challenging. Flink's checkpoint should be consistent among all stateful operators at the same time. For an *embedded* and *file-based* key value store like RocksDB, it is easier to implement by uploading files of specific time asynchronously.
Moreover if you want to store your state basically in memory, then why not using the HashMapStateBackend. It saves the overhead of serialization and deserialization and may achieve better performance compared with Redis I guess. Best, Zakelly On Tue, Jan 30, 2024 at 2:15 PM Chirag Dewan via user <user@flink.apache.org> wrote: > Hi, > > I was looking at the FLIP-254: Redis Streams Connector and I was > wondering if Flink ever considered Redis as a state backend? And if yes, > why was it discarded compared to RocksDB? > > If someone can point me towards any deep dives on why RocksDB is a better > fit as a state backend, it would be helpful. > > Thanks, > Chirag >
