it does and does not. say, scheduler heartbeat = 30 sec You will see a spiky cpu consumption graph every 30 seconds.
But we did not go that route and kept the scheduler heartbeat = 5 sec so that we do not lose time when a task is ready to run (I think there is another known bug here - tasks dont move from one queued -> running state even after "job heartbeat" ) On Tue, Mar 7, 2017 at 10:41 AM, Jason Chen <[email protected]> wrote: > Hi Harish, > Thanks for the fast response and feedback. > Yeah, I want to see the fix or more discussion ! > > BTW, I assume that, given your 30 dags, airflow runs fine after your > increase of heartbeat ? > The default is 5 secs. > > > Thanks. > Jason > > > On Tue, Mar 7, 2017 at 10:24 AM, harish singh <[email protected]> > wrote: > > > I had seen a similar behavior, a year ago, when we were are < 5 Dags. > Even > > then the cpu utilization was reaching 100%. > > One way to deal with this is - You could play with "heatbeat" numbers > (i.e > > increase heartbeat). > > But then you are introducing more delay to start jobs that are ready to > run > > (ready to be queued -> queued -> run) > > > > Right now, we have more than 30 dags (each with ~ 20-25 tasks) that runs > > every hour. > > We are giving airflow about 5-6 cores (which still seems less for > airflow). > > Also, for so many tasks every hour, our mem consumption is over 16G. > > All our tasks are basically doing "curl". So 16G seems too high. > > > > Having said that, I remember reading somewhere that there was a fix > coming > > for this. > > If not, I would definitely want to see more discussion on this. > > > > Thanks for opening this. I would love to hear on how people are working > > around this. > > > > > > > > > > > > On Tue, Mar 7, 2017 at 9:42 AM, Jason Chen <[email protected]> > > wrote: > > > > > Hi team, > > > > > > We are using airflow v1.7.1.3 and schedule about 50 dags (each dags is > > > about 10 to one hour intervals). It's with LocalExecutor. > > > > > > Recently, we noticed the RDS (MySQL 5.6.x with AWS) runs with ~100% > CPU. > > > I am wondering if airflow scheduler and webserver can cause high CPU > load > > > of MySQL, given ~50 dags? > > > I feel MySQL should be light load.. > > > > > > Thanks. > > > -Jason > > > > > >
