spoutnickgp opened a new issue #18096: URL: https://github.com/apache/airflow/issues/18096
### Apache Airflow version 2.1.3 (latest released) ### Operating System Ubuntu 20.04.3 LTS ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==2.2.0 apache-airflow-providers-cncf-kubernetes==2.0.2 apache-airflow-providers-ftp==2.0.1 apache-airflow-providers-google==5.1.0 apache-airflow-providers-http==2.0.1 apache-airflow-providers-imap==2.0.1 apache-airflow-providers-postgres==2.2.0 apache-airflow-providers-sftp==2.1.1 apache-airflow-providers-sqlite==2.0.1 apache-airflow-providers-ssh==2.1.1 ### Deployment Other Docker-based deployment ### Deployment details docker-compose version 1.29.2, build 5becea4c Docker version 20.10.8, build 3967b7d ### What happened When a major exception occurs in the scheduler (such as a RDBMS backend failure/restart) or basically anything triggering a failure of the JobScheduler, if the log_serving is on, then although the JobScheduler process is stopped, the exception cascades all the way up to the CLI, but the shutdown of the `serve_logs` process is never triggered [here](https://github.com/apache/airflow/blob/ff64fe84857a58c4f6e47ec3338b576125c4223f/airflow/cli/commands/scheduler_command.py#L72), causing the parent process to wait idle, therefore making any watchdog/restart of the process fail with a downed scheduler in such instances. ### What you expected to happen I expect the whole CLI `airflow scheduler` to exit with an error so it can be safely restarted immediately without any external intervention. ### How to reproduce Launch any scheduler from the CLI with the log serving on (no `--skip-serve-logs`), then shutdown the RDBMS. This will cause the scheduler to exit, but the process will still be running, which is not the expected behavior. ### Anything else I already have a PR to submit for a fix on this issue, care to take a look at it ? Will link it in the issue. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
