topherinternational opened a new pull request, #71283: URL: https://github.com/apache/airflow/pull/71283
This change causes the scheduler to start the local log server if the `LocalExecutor` is anywhere in the list of configured executors; previously, scheduler would start the log server only if LocalExecutor was the default (first) executor. In my org, we set up multiple executors on Airflow 2 with the `core.executor` config set to `KubernetesExecutor,LocalExecutor`. We put the `KubernetesExecutor` first as a fail-safe - in the event a task runs without specifying an executor, we want it to run in an isolated pod so it doesn't threaten resources on the scheduler or its locally-running tasks. After we upgraded to Airflow 3, we noticed: * Task logs would not display in the UI for running local tasks (logs would display as expected for finished local tasks, and running and finished k8s tasks) * Logs in the api-server would show a corresponding `Connection refused` error trying to connect to the scheduler on port 8793 * The scheduler was not actually listening on 8793, i.e. the local log server had not been started I examined the code and found that `scheduler_command` launches the log server only if `LocalExecutor` is the default executor (i.e., the first executor listed in the configuration). This PR solves the issue by examining all configured executor types instead of only the default when deciding whether to launch the log server. I've added two test cases to the existing parameterized `TestSchedulerCommand.test_serve_logs_on_scheduler()` method: * `"LocalExecutor,KubernetesExecutor"` passes before and after this fix, showing that the log server is launched when the local executor is the default in a multiple-executor setup * `"KubernetesExecutor,LocalExecutor"` fails before this fix and passes after it, showing that the log server is launched when the local executor is non-default in a multiple-executor setup --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [ ] Yes (please specify the tool below) <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
