boltonidze commented on issue #44759: URL: https://github.com/apache/airflow/issues/44759#issuecomment-2819750569
Hello! Is there any solution to the original issue? I'm also planning to run deferred tasks in a pool where they are accounted for. In case of success, I immediately mark the task as successful, but in case of failure, I want the on_failure_callback to trigger, which sends notifications to Slack. Currently, it only works if the task is executed on workers. Also, for some operators, in case of failure, I’d like to run some custom logic. Right now, there are no guarantees that the task won’t get stuck in the pool waiting for a slot and end up running only after the failure has already occurred. Maybe a possible solution would be the ability to change the pool of a task after the deferred task has finished on a triggerrer. Then I could simply have two pools — one where deferred tasks are considered as running, and one where they are not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
