RNHTTR commented on code in PR #40468:
URL: https://github.com/apache/airflow/pull/40468#discussion_r1670868964
##########
airflow/executors/base_executor.py:
##########
@@ -284,8 +289,11 @@ def trigger_tasks(self, open_slots: int) -> None:
self.log.info("queued but still running; attempt=%s
task=%s", attempt.total_tries, key)
continue
# Otherwise, we give up and remove the task from the queue.
- self.log.error(
- "could not queue task %s (still running after %d
attempts)", key, attempt.total_tries
+ self.task_context_logger.error(
+ "could not queue task %s (still running after %d
attempts)",
Review Comment:
I think this is still really confusing.
> Could not queue task %s as it is seen as still running after %d attempts
(tried for %d seconds). It looks like it was killed externally. Look for
external reasons why it has been killed (likely a bug or deployment issue).
* What could not queue the task?
* Seen as still running by what?
* What does externally mean?
* Why is this likely a bug or deployment issue?
I had a convo with @wolfier @collinmcnulty @seanmuth and others, and below
is what we landed on. IMO it's too long to share as a log message and should be
added and linked to
[troubleshooting.rst](https://github.com/apache/airflow/blob/main/docs/apache-airflow/troubleshooting.rst):
> **State mismatch between Airflow Metadata Database and Executor**
> This indicates that when the scheduler queried the Airflow database, it
observed that the task instance's status was `queued`, but the status according
to the executor was `running`. This mismatch persisted for several attempts. As
a result, Airflow will not attempt to queue the task. It's possible that
something has gone wrong in the executor, and the task may need to be cleared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
