Hi Jason, See this from the conf file:
# The scheduler can run multiple threads in parallel to schedule dags. # This defines how many threads will run. However airflow will never # use more threads than the amount of cpu cores available. max_threads = 2 So probably the no of processes you see are equal to no of processes your machine have. About the "*num_runs*" things, scheduler process will gracefully dies once it reaches to the said no of cycles. A cycle is nothing but a loop of iteration over all the dags present in your dags folder. Thanks, Sumit On Sun, Sep 4, 2016 at 8:28 AM, Jason Chen <[email protected]> wrote: > Hi airflow team, > > We setup airflow as a upstart service using the > suggestion here > https://github.com/apache/incubator-airflow/blob/master/ > scripts/upstart/airflow-scheduler.conf#L33 > > We set SCHEDULER_RUNS=0 (unlimit). > > We notice that using "ps aux | grep airflow", it indicates several > scheduler running as below... > I am thinking it's normal, because the scheduler executes for each run (and > there are multiple runs at the same time). > Is that right ? > Also, can you explain "Scheduler will restart after every ${SCHEDULER_RUNS} > runs" ? > > Thanks. > > Jason > > > ...... > ec2-user 25517 0.0 0.1 405372 56776 pts/0 Sl+ 02:53 0:00 > /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0 > ec2-user 25518 0.0 0.1 405372 56544 pts/0 S+ 02:53 0:00 > /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0 > ec2-user 25519 0.0 0.1 405372 56772 pts/0 Sl+ 02:53 0:00 > /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0 > ec2-user 25520 0.0 0.1 405372 56772 pts/0 Sl+ 02:53 0:00 > /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0 > ec2-user 25521 0.0 0.1 405372 56548 pts/0 S+ 02:53 0:00 > /usr/bin/python2.7 /usr/local/bin/airflow scheduler -n 0 > ........ >
