The scheduler constantly attempts to schedule tasks, interacting with the database and reloading DAG definition. In most larg-ish environments, burning up to a CPU to run the scheduler doesn't seem outrageous to me.
If you want to reduce the CPU load related to the scheduler check out SCHEDULER_HEARTBEAT_SEC and MAX_THREADS in the scheduler section of `airflow.cfg` Max On Sun, Jun 12, 2016 at 1:24 PM, harish singh <[email protected]> wrote: > Hi guys, > > We are running airflow (for about 3 months now) inside a docker container > on aws. > > I just did a docker stats to check whats going on. The cpu consumption is > huge. > We have around 15 DAGS. Only one DAG is turned ON. the remaining are OFF. > The DAG runs with a HOURLY schedule. > > Right now, airflow is consuming almost 1 complete core. > It seems there is some unnecessary spinning? > This doesnt look like the right behavior. > Is there a bug already filed for this? Or am not sure if there is something > incorrect in the way I am using the airflow configuration. > > CONTAINER CPU % MEM USAGE / LIMIT MEM % > NET I/O BLOCK I/O > CCC 68.17% 619.7 MB / 2.147 GB > 28.85% 1.408 GB / 939.4 MB 7.856 MB / 0 B > XXX 64.36% 619.4 MB / 2.147 GB > 28.84% 1.211 GB / 807.6 MB 7.856 MB / 0 B > > > Ariflow version 1.7.0 > > Airflow config: > > sql_alchemy_pool_size = 5 > sql_alchemy_pool_recycle = 3600 > parallelism = 8 > dag_concurrency = 8 > max_active_runs_per_dag = 8 > > > > Thanks, > > Harish >
