kaxil commented on pull request #10956: URL: https://github.com/apache/airflow/pull/10956#issuecomment-702997850
Instructions to run this PR with docker-compose using CeleryExecutor with Multiple Schedulers: - Clone the repo and checkout the branch ``` git clone --single-branch --branch scheduler-ha-astro https://github.com/astronomer/airflow.git scheduler-ha-airflow cd scheduler-ha-airflow ``` - Run the docker-compose files: **Postgres**: - Run the following command to run the scheduler container (and it's dependencies) that would also take care of running `airflow db upgrade` (creating required tables) ``` docker-compose -f docker-compose-postgres.yml run scheduler ``` - You can run the above command with `-d` too to run it in background. Or just Ctrl + C after the migrations are run. - Scale Up the number of Schedulers to 2 and workers to 4 ``` docker-compose -f docker-compose-postgres.yml up --scale worker=4 --scale scheduler=2 ``` **MySQL 8**: Same as Postgres but no need to specify the docker-compose file as the default is `docker-compose.yml` (which contains MySQL code) ``` docker-compose run scheduler ``` - You can run the above command with `-d` too to run it in background. Or just Ctrl + C after the migrations are run. - Scale Up the number of Schedulers to 2 and workers to 4 ``` docker-compose up --scale worker=4 --scale scheduler=2 ``` **MySQL 5.7**: ``` docker-compose -f docker-compose-mysql57.yml up --scale worker=4 ``` **Note**: Running Multiple Schedulers is not supported with MySQL 5.7. You can also try the escape hatch to not use locking mechanism by adding the following environment variable to the docker-compose file: ``` AIRFLOW__SCHEDULER__USE_ROW_LEVEL_LOCKING: "False" ``` Trigger some example dags from the Webserver (Port: 8080) to see it in action or put your own dags under the `dags` directory (`scheduler-ha-airflow/dags`) -- that directory is mounted on all containers. Flower UI is exposed on http://localhost:5555/ ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
