jedcunningham commented on issue #16888: URL: https://github.com/apache/airflow/issues/16888#issuecomment-876779262
@duc-luong-tnsl, we watch pods with this label: https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L129 In your scheduler log, right before the traceback, what pod name was logged? Does it looks like a kubernetesexecutor worker (e.g. dagidtaskid.random)? https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L147 The only time I've seen this previously is when incorrect labels were added via `pod_mutation_hook` causing KPO pods to be considered as workers in the worker adoption logic, which then lead to this traceback. See https://github.com/astronomer/airflow-chart/pull/231. Do you have a `pod_mutation_hook` that sets `kubernetes_executor`? Or happen to use the https://github.com/astronomer/airflow-chart chart? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
