I had seen a similar behavior, a year ago, when we were are < 5 Dags. Even then the cpu utilization was reaching 100%. One way to deal with this is - You could play with "heatbeat" numbers (i.e increase heartbeat). But then you are introducing more delay to start jobs that are ready to run (ready to be queued -> queued -> run)
Right now, we have more than 30 dags (each with ~ 20-25 tasks) that runs every hour. We are giving airflow about 5-6 cores (which still seems less for airflow). Also, for so many tasks every hour, our mem consumption is over 16G. All our tasks are basically doing "curl". So 16G seems too high. Having said that, I remember reading somewhere that there was a fix coming for this. If not, I would definitely want to see more discussion on this. Thanks for opening this. I would love to hear on how people are working around this. On Tue, Mar 7, 2017 at 9:42 AM, Jason Chen <[email protected]> wrote: > Hi team, > > We are using airflow v1.7.1.3 and schedule about 50 dags (each dags is > about 10 to one hour intervals). It's with LocalExecutor. > > Recently, we noticed the RDS (MySQL 5.6.x with AWS) runs with ~100% CPU. > I am wondering if airflow scheduler and webserver can cause high CPU load > of MySQL, given ~50 dags? > I feel MySQL should be light load.. > > Thanks. > -Jason >
