Hi, I deployed Flink 1.10 standalone in a cluster with 4 machines 8 cores each. Then I configured each machine to have 8 Task Slots and parallelism default of 8. taskmanager.numberOfTaskSlots: 8 parallelism.default: 8 I want to run my stream app with a parallelism of 16 for each subtask. But not having more than 8 subtasks in one TaskManager because Flink will make them share memory [1]. I suppose that I can deploy half of the subtasks in one machine and the second half in another machine. Is it correct?
Then I deployed a program using "flink run -p 16 myApp.jar" and I was monitoring it with Prometheus + Grafana and Flink created 16 subtasks in two nodes. I mean that I am seeing 32 subtasks of each operator that I am using. I think I was supposed to see only 16 of each operator. Is there something wrong with my configuration? Additional to that I also tried to use .setParallelism(16) but I got the same result. 32 subtasks of the same operator. Thanks, Felipe [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-numberoftaskslots -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com
