FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to
be a bottleneck. Perhaps due to high frequency of updating state components.I
replaced it with Java 8 ConcurrentHashmaps and it settled down
tremendously.Amir-
From: Ovidiu-Cristian MARCU <[email protected]>
To: [email protected]
Sent: Tuesday, October 25, 2016 12:40 AM
Subject: Re: [FLINK-3035] Redis as State Backend
Thank you!
Best,
Ovidiu
> On 24 Oct 2016, at 16:11, Aljoscha Krettek <[email protected]> wrote:
>
> Hi,
> regarding RocksDB, yes this is possible because RocksDB is essentially only
> used as an out-of-core hash table. When checkpointing we write everything
> from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB
> instance from the data in HDFS.
>
> Cheers,
> Aljoscha
>
> On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu <
> [email protected]> wrote:
>
>> Hi
>>
>> I missed your reply, thank you for feedback.
>> Agree with 1, that will be possible only with Ramcloud.
>> It is clear the second point.
>>
>> A short question: if you checkpoint the operator's state in hdfs I assume
>> that on failure you are
>> restarting the operator's tasks on other nodes, is that possible with
>> RocksDB?
>>
>> Best,
>> Ovidiu
>>
>> -----Original Message-----
>> From: Aljoscha Krettek [mailto:[email protected]]
>> Sent: Monday, October 17, 2016 2:51 PM
>> To: [email protected]
>> Subject: Re: [FLINK-3035] Redis as State Backend
>>
>> Hi,
>> there are two basic ideas for implementing a StateBackend based on Redis:
>> 1. Rely on Redis to keep the state, use nothing else.
>> 2. Use Redis to keep the state and checkpoint to some distributed file
>> system (such as HDFS) when checkpointing
>>
>> The first idea seems unwise because Redis is not a "strongly consistent
>> distributed data store" as Elias pointed out on the issue. The second Idea
>> is problematic because there is no easy way to read all state for a given
>> Flink operator from a running Redis instance to store it in HDFS. That's
>> what I was getting at in my comment.
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu <
>> [email protected]> wrote:
>>
>>> Hi
>>>
>>> Can you please expand the last comment:
>>>
>>> "I think, however, that for other reasons we will probably not be able
>>> to implement this well. The problem is that we have to somehow get at
>>> the state in redis for checkpointing. And if we use only one Redis
>>> instance for all states then this will be problematic." - Aljoscha
>>> Krettek
>>>
>>> Any other update on this issue will help, not clear the status.
>>>
>>> Best,
>>> Ovidiu
>>>
>>>
>>
