nt: Sunday, October 29, 2017 6:05 PM
Subject: Re: Capacity Planning For Large State in YARN Cluster
Well you can only performance test it beforehand in different scenarios with
different configurations.
I am not sure what exactly your state holds (eg how many objects etc), but if
it is Java ob
> Thanks, Ashish
>
> --
> *From:* Jörn Franke <jornfra...@gmail.com>
> *To:* Ashish Pokharel <ashish...@yahoo.com>
> *Cc:* Till Rohrmann <trohrm...@apache.org>; user <user@flink.apache.org>
> *Sent:* Sunday, October 29, 2017 6:0
From: Jörn Franke <jornfra...@gmail.com>
To: Ashish Pokharel <ashish...@yahoo.com>
Cc: Till Rohrmann <trohrm...@apache.org>; user <user@flink.apache.org>
Sent: Sunday, October 29, 2017 6:05 PM
Subject: Re: Capacity Planning For Large State in YARN Cluster
Well you c
Well you can only performance test it beforehand in different scenarios with
different configurations.
I am not sure what exactly your state holds (eg how many objects etc), but if
it is Java objects then 3 times might be a little bit low (depends also how you
initially tested state size) -
Hi Till,
I got the same feedback from Robert Metzger over in Stackflow. I have switched
my app to use RocksDB and as yes, it did stabilize the app :)
However, I am still struggling with how to map out my TMs and JMs memory,
number of slots per TMs etc. Currently I am using 60 slots with 10
Hi Ashish,
what you are describing should be a good use case for Flink and it should
be able to run your program.
When you are seeing a GC overhead limit exceeded error, then it means that
Flink or your program are creating too many/too large objects filling up
the memory in a short time. I
