Re: Flink: For terabytes of keyed state.

Hi Gowri Please let us know if you meet any problem~ Best, Congxian Gowri Sundaram 于2020年5月6日周三 下午1:53写道： > Hi Congxian, > Thank you so much for your response! We will go ahead and do a POC to test > out how Flink performs at scale. > > Regards, > - Gowri > > On Wed, May 6, 2020 at 8:34 AM

Re: Flink: For terabytes of keyed state.

Hi Congxian, Thank you so much for your response! We will go ahead and do a POC to test out how Flink performs at scale. Regards, - Gowri On Wed, May 6, 2020 at 8:34 AM Congxian Qiu wrote: > Hi > > From my experience, you should care the state size for a single task(not > the whole job state

Re: Flink: For terabytes of keyed state.

Hi >From my experience, you should care the state size for a single task(not the whole job state size), the download speed for single thread is almost 100 MB/s (this may vary in different env), and I do not have much performance for loading state into RocksDB(we use an internal KV store in my

Re: Flink: For terabytes of keyed state.

Hi Congxian, Thank you so much for your response, that really helps! >From your experience, how long does it take for Flink to redistribute terabytes of state data on node addition / node failure. Thanks! On Sun, May 3, 2020 at 6:56 PM Congxian Qiu wrote: > Hi > > 1. From my experience, Flink

Re: Flink: For terabytes of keyed state.

Hi 1. From my experience, Flink can support such big state, you can set appropriate parallelism for the stateful operator. for RocksDB you may need to care about the disk performance. 2. Inside Flink, the state is separated by key-group, each task/parallelism contains multiple key-groups. Flink

Flink: For terabytes of keyed state.

Hello all, We have read in multiple sources that Flink has been used for use cases with terabytes of application state. We are considering using Flink for a similar use case