SamWheating commented on issue #20788: URL: https://github.com/apache/airflow/issues/20788#issuecomment-1010169302
OK, so looking into this a bit, the scheduler will log a warning if a task is unschedulable due to non-existent pool: https://github.com/apache/airflow/blob/905baf9fa5402ccc062536915fd1911d812f625b/airflow/jobs/scheduler_job.py#L335-L339 This warning is also visible in the TaskInstance Details UI:  And then it will remain in `queued` state indefinitely (or until it times out I suppose) It would be really simple to just mark the tasks as failed after logging something like `Tasks using non-existent pool '%s' being marked as Failed`, but this might be a worse user experience as it leaves no logs or visible warnings about why the task failed (other than the scheduler logs, which are not easily accessible to most users) With this in mind, does anyone have another idea for how to prevent these tasks from clogging the scheduler, or should we just consider this to be intended behaviour? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
