Thanks, how big was your state (GBs)? Can you share your benchmark/s? Best, Ovidiu
-----Original Message----- From: amir bahmanyari [mailto:amirto...@yahoo.com.INVALID] Sent: Tuesday, October 25, 2016 7:24 PM To: dev@flink.apache.org Subject: Re: [FLINK-3035] Redis as State Backend FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU <ovidiu-cristian.ma...@inria.fr> To: dev@flink.apache.org Sent: Tuesday, October 25, 2016 12:40 AM Subject: Re: [FLINK-3035] Redis as State Backend Thank you! Best, Ovidiu > On 24 Oct 2016, at 16:11, Aljoscha Krettek <aljos...@apache.org> wrote: > > Hi, > regarding RocksDB, yes this is possible because RocksDB is essentially > only used as an out-of-core hash table. When checkpointing we write > everything from RocksDB to HDFS. When restoring we repopulate an empty > local RocksDB instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > ovidiu.cristian.ma...@huawei.com> wrote: > >> Hi >> >> I missed your reply, thank you for feedback. >> Agree with 1, that will be possible only with Ramcloud. >> It is clear the second point. >> >> A short question: if you checkpoint the operator's state in hdfs I >> assume that on failure you are restarting the operator's tasks on >> other nodes, is that possible with RocksDB? >> >> Best, >> Ovidiu >> >> -----Original Message----- >> From: Aljoscha Krettek [mailto:aljos...@apache.org] >> Sent: Monday, October 17, 2016 2:51 PM >> To: dev@flink.apache.org >> Subject: Re: [FLINK-3035] Redis as State Backend >> >> Hi, >> there are two basic ideas for implementing a StateBackend based on Redis: >> 1. Rely on Redis to keep the state, use nothing else. >> 2. Use Redis to keep the state and checkpoint to some distributed >> file system (such as HDFS) when checkpointing >> >> The first idea seems unwise because Redis is not a "strongly >> consistent distributed data store" as Elias pointed out on the issue. >> The second Idea is problematic because there is no easy way to read >> all state for a given Flink operator from a running Redis instance to >> store it in HDFS. That's what I was getting at in my comment. >> >> Cheers, >> Aljoscha >> >> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < >> ovidiu.cristian.ma...@huawei.com> wrote: >> >>> Hi >>> >>> Can you please expand the last comment: >>> >>> "I think, however, that for other reasons we will probably not be >>> able to implement this well. The problem is that we have to somehow >>> get at the state in redis for checkpointing. And if we use only one >>> Redis instance for all states then this will be problematic." - >>> Aljoscha Krettek >>> >>> Any other update on this issue will help, not clear the status. >>> >>> Best, >>> Ovidiu >>> >>> >>
