Re: Flink long-running YARN configuration

> there. >> >> /cc Robert >> >> Greetings, >> Stephan >> >> >> On Thu, Aug 25, 2016 at 5:02 PM, Foster, Craig <foscr...@amazon.com> >> wrote: >> >>> I'm trying to understand Flink YARN configuration. T

Re: Flink long-running YARN configuration

25, 2016 at 5:02 PM, Foster, Craig <foscr...@amazon.com> > wrote: > >> I'm trying to understand Flink YARN configuration. The flink-conf.yaml >> file is supposedly the way to configure Flink, except when you launch Flink >> using YARN since that's determined for the A

Re: Flink long-running YARN configuration

from there. /cc Robert Greetings, Stephan On Thu, Aug 25, 2016 at 5:02 PM, Foster, Craig <foscr...@amazon.com> wrote: > I'm trying to understand Flink YARN configuration. The flink-conf.yaml > file is supposedly the way to configure Flink, except when you launch Flink > using YA

Re: Yarn configuration

OK thanks Robert you have been very clear now! :) just one question, more related on emr than to flink, if i cannot run anything on the EMR master, then is it useful to allocate a big machine (8 core, 30GB) on it? I thought it was the jm but it is not Il giorno 27/lug/2015, alle ore 14:56,

Re: Yarn configuration

Hi Michele, I'm happy that you got it to run the way you want. I guess services such as the HDFS NameNode and YARNs ResourceManager are running on the master. I don't know what you are doing on the cluster, but I suspect it is for experimentation only. As long as you are not maintaining a huge

Re: Yarn configuration

Hi Michele, the 10506 MB refer to the size of Flink's managed memory whereas the 20992 MB refer to the total amount of TM memory. At start-up, the TM allocates a fraction of the JVM memory as byte arrays and manages this portion by itself. The remaining memory is used as regular JVM heap for TM

Re: Yarn configuration

Hi Fabian, thanks for your reply so you flink is using about 50% of memory for itself right? anyway now I am running an EMR with 1 master and 5 core all of them are m3.2xlarge with 8 cores and 30GB of memory I would like to run flink on yarn with 40 slots on 5 tm with the maximum available

Re: Yarn configuration

Hi Robert, thanks for answering, today I have been able to try again: no in an EMR configuration with 1 master and 5 core I have 5 active node in the resource manager…sounds strange to me: ganglia shows 6 nodes and 1 is always offload the total amount of memory is 112.5GB that is actually 22.5

Re: Yarn configuration

Hi Michele, configuring a YARN cluster to allocate all available resources as good as possible is sometimes tricky, that is true. We are aware of these problems and there are actually the following two JIRAs for this: https://issues.apache.org/jira/browse/FLINK-937 (Change the YARN Client to

Yarn configuration

Hi everybody, i need a help on how to configure a yarn cluster I tried a lot of conf but none of them was correct We have a cluster on amazon emr let's say 1manager+5worker all of them are m3.2xlarge then 8 core each and 30 GB of RAM each What is a good configuration for such cluster? I would

Re: flink on yarn configuration

Hi Paul, when you run your Flink cluster with YARN then we cannot give the full amount of the allocated container memory to Flink. The reason is that YARN itself needs some of the memory as well. Since YARN is quite strict with containers which exceed their memory limit (the container is