Thanks a lot. On Tue, Sep 11, 2018, 6:25 PM ramandu...@gmail.com <ramandu...@gmail.com> wrote:
> We use max_threads = number of scheduler cores. > > On 2018/09/11 09:49:53, Chandu Kavar <ccka...@gmail.com> wrote: > > Thanks Raman, > > > > Understood. > > > > We have around 500 DAGs. What value do you suggest for max_threads? > > > > On Tue, Sep 11, 2018, 5:44 PM ramandu...@gmail.com <ramandu...@gmail.com > > > > wrote: > > > > > Hi Chandu, > > > How many dag files are there on the scheduler. As per my understanding > > > scheduler processes each file to trigger any dag/task run. It spawns > number > > > of processes equivalent to "max_threads" count to parallelize file > > > processing. So you can try by increasing airflow config's max_threads > count. > > > > > > Thanks, > > > Raman Gupta > > > > > > On 2018/09/11 07:11:23, Chandu Kavar <ccka...@gmail.com> wrote: > > > > Hi All, > > > > > > > > We are seeing there is a delay between triggering the next task. > Please > > > > find the attached screenshot. > > > > > > > > First and the last task are very small. So, you can see the tiny > object. > > > > > > > > [image: image.png] > > > > Do you suggest any airflow config can resolve this problem? > > > > > > > > Thanks, > > > > Chandu > > > > > > > > > >
