Hi Julio, When you start the Flink YARN session, you have to specify the number of TaskManagers and number of slots per TaskManager. There is currently no officially supported way to add more TaskManagers to a long running YARN session. We are aware of this limitation, and there are ongoing development efforts to make the Flink Cluster dynamically allocate new YARN containers when needed (FLIP-6) [1]. We are aiming at shipping this feature with the Flink 1.5 release [2]. As a workaround, you can restore your jobs from savepoints on a bigger cluster.
In a standalone cluster it is possible to add TaskManagers to a running cluster [3]. Best, Gary [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 [2] https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html [3] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/cluster_setup.html#adding-jobmanagertaskmanager-instances-to-a-cluster On Thu, Feb 1, 2018 at 5:09 PM, Julio Biason <julio.bia...@azion.com> wrote: > Hi, > > I'm wodering, is there a way to make Flink understand that I added a new > worker machine on yarn to increase the tasks slots? > > Right now, we are just exploring how to run Flink on Yarn and, so far, we > managed to create a small Hadoop/Yarn cluster (3 machines, 1 master and 2 > workers) and start Flink on it. > > But say I want to extend this cluster to add more machines; in this case, > simply adding more machines to the Hadoop cluster will work? > > -- > *Julio Biason*, Sofware Engineer > *AZION* | Deliver. Accelerate. Protect. > Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 > <callto:+5551996209291>*99907 0554* >