Hi, Mu, AFAIK, this feature is added to 1.9.2. If you use 1.9.0, would you like to upgrade your Flink distribution?
Best, Yangze Guo On Tue, Jul 7, 2020 at 8:33 PM Mu Kong <kong.mu....@gmail.com> wrote: > > Hi, Guo, > > Thanks for helping out. > > My application has a kafka source with 60 subtasks(parallelism), and we have > 15 task managers with 15 slots on each. > > Before I applied the cluster.evenly-spread-out-slots, meaning it is set to > default false, the operator 'kafka source" has 11 subtasks allocated in one > single task manager, > while the remaining 49 subtasks of "kafka source" distributed to the > remaining 14 task managers. > > After I set cluster.evenly-spread-out-slots to true, the 60 subtasks of > "kafka source" were allocated to only 4 task managers, and they took 15 slots > on each of these 4 TMs. > > What I thought is that this config will make the subtasks of one operator > more evenly spread among the task managers, but it seems it made them > allocated in the same task manager as much as possible. > > The version I'm deploying is 1.9.0. > > Best regards, > Mu > > On Tue, Jul 7, 2020 at 7:10 PM Yangze Guo <karma...@gmail.com> wrote: >> >> Hi, Mu, >> >> IIUC, cluster.evenly-spread-out-slots would fulfill your demand. Why >> do you think it does the opposite of what you want. Do you run your >> job in active mode? If so, cluster.evenly-spread-out-slots might not >> work very well because there could be insufficient task managers when >> request slot from ResourceManager. This has been discussed in >> https://issues.apache.org/jira/browse/FLINK-12122 . >> >> >> Best, >> Yangze Guo >> >> On Tue, Jul 7, 2020 at 5:44 PM Mu Kong <kong.mu....@gmail.com> wrote: >> > >> > Hi community, >> > >> > I'm running an application to consume data from kafka, and process it then >> > put data to the druid. >> > I wonder if there is a way where I can allocate the data source consuming >> > process evenly across the task manager to maximize the usage of the >> > network of task managers. >> > >> > So, for example, I have 15 task managers and I set parallelism for the >> > kafka source as 60, since I have 60 partitions in kafka topic. >> > What I want is flink cluster will put 4 kafka source subtasks on each task >> > manager. >> > >> > Is that possible? I have gone through the document, the only thing we >> > found is >> > >> > cluster.evenly-spread-out-slots >> > >> > which does exact the opposite of what I want. It will put the subtasks of >> > the same operator onto one task manager as much as possible. >> > >> > So, is some kind of manual resource allocation available? >> > Thanks in advance! >> > >> > >> > Best regards, >> > Mu
