Hi Harish, Thanks for the fast response and feedback. Yeah, I want to see the fix or more discussion !
BTW, I assume that, given your 30 dags, airflow runs fine after your increase of heartbeat ? The default is 5 secs. Thanks. Jason On Tue, Mar 7, 2017 at 10:24 AM, harish singh <[email protected]> wrote: > I had seen a similar behavior, a year ago, when we were are < 5 Dags. Even > then the cpu utilization was reaching 100%. > One way to deal with this is - You could play with "heatbeat" numbers (i.e > increase heartbeat). > But then you are introducing more delay to start jobs that are ready to run > (ready to be queued -> queued -> run) > > Right now, we have more than 30 dags (each with ~ 20-25 tasks) that runs > every hour. > We are giving airflow about 5-6 cores (which still seems less for airflow). > Also, for so many tasks every hour, our mem consumption is over 16G. > All our tasks are basically doing "curl". So 16G seems too high. > > Having said that, I remember reading somewhere that there was a fix coming > for this. > If not, I would definitely want to see more discussion on this. > > Thanks for opening this. I would love to hear on how people are working > around this. > > > > > > On Tue, Mar 7, 2017 at 9:42 AM, Jason Chen <[email protected]> > wrote: > > > Hi team, > > > > We are using airflow v1.7.1.3 and schedule about 50 dags (each dags is > > about 10 to one hour intervals). It's with LocalExecutor. > > > > Recently, we noticed the RDS (MySQL 5.6.x with AWS) runs with ~100% CPU. > > I am wondering if airflow scheduler and webserver can cause high CPU load > > of MySQL, given ~50 dags? > > I feel MySQL should be light load.. > > > > Thanks. > > -Jason > > >
