ephraimbuddy commented on issue #16625: URL: https://github.com/apache/airflow/issues/16625#issuecomment-902296823
> @ephraimbuddy So the metadata database is seeing the task as failed (as was reported by the executor) so that's what's showing up in the UI, but the scheduler still thinks it's queued, so it never attempts to retry? Am I understanding the behavior correctly? > > If so, making sure that #15929 includes the task retrying if it has retries left will be key. Otherwise, will the behavior difference even be noticeable to the user? I think retrying is a lesser evil to getting stuck in queued. That was why I added #15929 which will be released in 2.1.3 The problem is when the executor reports that this task has failed and the scheduler sees it as queued, without #15929 it gets stuck in queued(even in the UI) and at times in up_for_retry(if it has retry) but never run again. It's also failed in some cases as @luqic said. But if it's stuck in `queued`, the task has to be cleared as @mmazek said above before it'd run again. See also https://github.com/apache/airflow/issues/13542. So without #15929, the task state would be set in` up_for_retry` at times as I found out but won't be run again. I still check for other possible solutions and your suggestion is worth trying -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
