whynick1 opened a new issue, #49244:
URL: https://github.com/apache/airflow/issues/49244
### Apache Airflow version
2.10.5
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
The Kubernetes API server's response body is not always guaranteed to be
JSON. For example, in the case of a 429 response, the body may be a plain
string (see example below).
```
kubernetes.client.exceptions.ApiException: (429)
Reason: Too Many Requests
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain;
charset=utf-8', 'Retry-After': '4', ...})
HTTP response body: Too many requests, please try again later.
```
This leads to unhandled exception in Airflow scheduler
([code](https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py#L365)).
```
except ApiException as e:
body = json.loads(e.body)
```
, causing scheduler pod to restart.
### What you think should happen instead?
Wrap the json.loads in a try-except to guard against malformed or non-JSON
bodies.
```
except ApiException as e:
try:
body = json.loads(e.body)
except json.JSONDecodeError:
# If the body is a string (e.g., in a 429 error), it can't be parsed
as JSON.
# Use the body directly as the message instead.
body = {"message": e.body}
```
### How to reproduce
1. Configure an Airflow deployment using the KubernetesExecutor or a setup
that interacts with the Kubernetes API (e.g., KubernetesPodOperator or the
scheduler using the K8s client).
2. Apply rate limiting on the Kubernetes API server (or simulate it via a
proxy or ingress controller) so that repeated API calls trigger a 429 Too Many
Requests response in plain text body.
3. Trigger a task run
4. See `json.decoder.JSONDecodeError` error in Airflow scheduler, and
scheduler restarts.
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
