Solved! that was because I was using slotSharingGroup() in all operators to ensure that they stay in the same task slot. I guess Flink was creating dummy operators to ensure that.
Thanks anyway. -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Mon, May 25, 2020 at 5:54 PM Felipe Gutierrez <felipe.o.gutier...@gmail.com> wrote: > > Hi, > > I deployed Flink 1.10 standalone in a cluster with 4 machines 8 cores > each. Then I configured each machine to have 8 Task Slots and > parallelism default of 8. > taskmanager.numberOfTaskSlots: 8 > parallelism.default: 8 > I want to run my stream app with a parallelism of 16 for each subtask. > But not having more than 8 subtasks in one TaskManager because Flink > will make them share memory [1]. I suppose that I can deploy half of > the subtasks in one machine and the second half in another machine. Is > it correct? > > Then I deployed a program using "flink run -p 16 myApp.jar" and I was > monitoring it with Prometheus + Grafana and Flink created 16 subtasks > in two nodes. I mean that I am seeing 32 subtasks of each operator > that I am using. I think I was supposed to see only 16 of each > operator. Is there something wrong with my configuration? > Additional to that I also tried to use .setParallelism(16) but I got > the same result. 32 subtasks of the same operator. > > Thanks, > Felipe > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-numberoftaskslots > > -- > -- Felipe Gutierrez > -- skype: felipe.o.gutierrez > -- https://felipeogutierrez.blogspot.com
