kristoffern commented on issue #19699: URL: https://github.com/apache/airflow/issues/19699#issuecomment-976612263
@ephraimbuddy The `external_executor_id` has a value attached to it, but that ID can't be found in the `celery_taskmeta` table. I did confirm that other `external_executor_id` that were successful had matching entries in the `celery_taskmeta` table :-) It's almost as if (and this is just speculation) a task is created and a row in `external_executor_id` is created, and somewhere before the system gets an ID back from the worker then that worker is killed by autopilot in GCP. And therefor no ID is added to `celery_taskmeta`. But again this last part is just speculation as I haven't found a 100% deterministic way of reproducing it. 9 mornings out of 10 though we have this problem, and most of the time tasks that are stuck in "queued" all have roughly the same starting time within a second or so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
