Re: State management and heap usage

[TechnoMage]

Thank you.

Michael

> On Apr 12, 2018, at 2:45 AM, Gary Yao  wrote:
> 
> Hi Michael,
> 
> You can configure the default state backend by setting state.backend in
> flink-conf.yaml, or you can configure it per job [1]. The default state 
> backend
> is "jobmanager" (MemoryStateBackend), which stores state and checkpoints on 
> the
> Java heap. RocksDB must be explicitly enabled, e.g., by setting state.backend 
> to
> "rocksdb".
> 
> Best,
> Gary
> 
> [1] 
> https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#configuring-a-state-backend
>  
> 
> 
> On Wed, Apr 11, 2018 at 11:04 PM, TechnoMage  > wrote:
> I am pretty new to flink and have an initial streaming job working both 
> locally and remotely.  But, both ways if the data volume is too high it runs 
> out of heap.  I am using RichMapFunction to process multiple streams of data. 
>  I assumed Flink would manage keeping state in ram when possible, and spill 
> to RocksDB when it exceeded heap.
> 
> Is this correct?  If so are there configs I need to set to enable or tune 
> this so it can run within a fixed memory size?
> 
> Michael
> 

Re: State management and heap usage

[Gary Yao]

Hi Michael,

You can configure the default state backend by setting state.backend in
flink-conf.yaml, or you can configure it per job [1]. The default state
backend
is "jobmanager" (MemoryStateBackend), which stores state and checkpoints on
the
Java heap. RocksDB must be explicitly enabled, e.g., by setting
state.backend to
"rocksdb".

Best,
Gary

[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#configuring-a-state-backend

On Wed, Apr 11, 2018 at 11:04 PM, TechnoMage  wrote:

> I am pretty new to flink and have an initial streaming job working both
> locally and remotely.  But, both ways if the data volume is too high it
> runs out of heap.  I am using RichMapFunction to process multiple streams
> of data.  I assumed Flink would manage keeping state in ram when possible,
> and spill to RocksDB when it exceeded heap.
>
> Is this correct?  If so are there configs I need to set to enable or tune
> this so it can run within a fixed memory size?
>
> Michael