Hi Airflow team, We are using Airflow with LocalExecutor and it works great. We are moving toward to use CeleryExecutor and have couple of questions. I searched the posts and cannot find some answers.
We have 3 airflow worker nodes and uses Redis as broker. (1) How airflow worker determines the next task should be dispatched to ? Is it based on the average load of the nodes ? How long the task runs previously ? (2) In my settings of 3 nodes, I have webserver, scheduler and worker (w/ Celery) running on each node. Is it a recommended setting (i.e., each service runs on each node) ? (3) When a task runs and fails, I have a on_failure_callback. Will the on_failure_callback runs on the same node as the task ? Or, the executor is based on task level (seems so) ? (4) There is setting for the queue (default_queue = default) in airflow.cfg. Say, I want to use different queues to control which worker should use which queue. I think just change default_queue = other_queue. Right ? Should I do any additional Celery related setting ? (5) If all workers are done and schedulers are still running, will the tasks be queued ? (6) If we turned of the tasks, while there are tasks in the queue previously, will it kill the un-needed tasks in the queue ? Thanks. Jason
