GitHub user kardamk added a comment to the discussion: Airflow worker pod not
able to fetch connections information during its execution
The issue continues to persist, and I carried out a few additional
troubleshooting steps to narrow down the root cause:
1. Extracted the API authorization token from the worker pod.
2. Executed a curl request using the Bearer token and received a successful
response from the API server. One thing that stood out is that the connection
configuration appears to be missing the host value:
{
"conn_id":"aws_default",
"conn_type":"aws",
"host":"",
"schema":"",
"login":"xxx",
"password":"xxx",
"port":null,
"extra":"{\"aws_access_key_id\": \"xxx\", \"aws_secret_access_key\": \"xxx\",
\"endpoint_url\": \"https://objectstore\"}";
}
3, Based on the successful curl request from the worker pod, networking or
connectivity issues between the worker and API server pods on OpenShift do not
appear to be the primary cause. If there were a connectivity problem, the
authenticated API call should not have succeeded.
4. To rule out remote logging as a contributing factor, I temporarily disabled
it and tested again. While investigating whether the call to
http://airflow-api-service:8080/execution/connections/aws_default was causing
the failure, I observed that the http://airflow-api-service:8080/execution/run
endpoint also returned a 503 error. This suggests the problem extends beyond
connection retrieval alone.
# Responses to the suggested checks:
1. The curl request succeeds from the worker pod when using the correct Bearer
token.
3. There are no corresponding connection-fetch logs in the API server logs.
This could potentially indicate an issue in the communication path before the
request reaches the API server, although I have not been able to confirm this
yet.
3. The AIRFLOW__CORE__EXECUTION_API_SERVER_URL value is configured correctly.
4. The service selector, endpoints, and targetPort all correctly resolve to the
Airflow 3 API server, and the API server pods are healthy and in a Ready state.
5. There are no routes or ingress resources in front of the API server; the
worker pod is accessing it directly through the internal service.
At this stage, I am still trying to determine the underlying cause. My current
suspicion is that the issue may be related either to the remote logging
configuration, which appears to be invoked before the worker can execute its
task, or to the JWT-based authentication mechanism used for API authorization.
The fact that both the connection lookup and execution endpoints are returning
failures despite direct authenticated access succeeding makes the behavior
somewhat difficult to reconcile.
GitHub link:
https://github.com/apache/airflow/discussions/69733#discussioncomment-18277941
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
