Hi, I have increased the number of slots available but the Job is not using all the slots but runs into this approximate 18000 Tasks limit. Looking into the source code, it seems to be opening file - https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java#L203 So, do I have to tune the ulimit or something similar at the Ubuntu O/S level to increase number of tasks available ? What I am confused about is the ulimit is per machine but the ExecutionGraph is across many machines ? Please pardon my ignorance here. Does number of tasks equate to number of open files. I am using 15 slots per TaskManager on AWS m5.4xlarge which has 16 vCPUs.
TIA. On Tue, May 19, 2020 at 3:22 PM Vijay Balakrishnan <bvija...@gmail.com> wrote: > Hi, > > Flink Dashboard UI seems to show tasks having a hard limit for Tasks > column around 18000 on a Ubuntu Linux box. > I kept increasing the number of slots per task manager to 15 and number of > slots increased to 705 but the slots to tasks > stayed at around 18000. Below 18000 tasks, the Flink Job is able to start > up. > Even though I increased the number of slots, it still works when 312 slots > are being used. > > taskmanager.numberOfTaskSlots: 15 > > What knob can I tune to increase the number of Tasks ? > > Pls find attached the Flink Dashboard UI. > > TIA, > >
