Hi, Mu, IIUC, cluster.evenly-spread-out-slots would fulfill your demand. Why do you think it does the opposite of what you want. Do you run your job in active mode? If so, cluster.evenly-spread-out-slots might not work very well because there could be insufficient task managers when request slot from ResourceManager. This has been discussed in https://issues.apache.org/jira/browse/FLINK-12122 .
Best, Yangze Guo On Tue, Jul 7, 2020 at 5:44 PM Mu Kong <kong.mu....@gmail.com> wrote: > > Hi community, > > I'm running an application to consume data from kafka, and process it then > put data to the druid. > I wonder if there is a way where I can allocate the data source consuming > process evenly across the task manager to maximize the usage of the network > of task managers. > > So, for example, I have 15 task managers and I set parallelism for the kafka > source as 60, since I have 60 partitions in kafka topic. > What I want is flink cluster will put 4 kafka source subtasks on each task > manager. > > Is that possible? I have gone through the document, the only thing we found is > > cluster.evenly-spread-out-slots > > which does exact the opposite of what I want. It will put the subtasks of the > same operator onto one task manager as much as possible. > > So, is some kind of manual resource allocation available? > Thanks in advance! > > > Best regards, > Mu
