unixengineer opened a new issue, #71010:
URL: https://github.com/apache/airflow/issues/71010
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.1 (code path still present on `main` as of this report)
### What happened and how to reproduce it?
## Summary
`VariableBody.value` is typed as `JsonValue`, so the Variables REST API
accepts **any** JSON type (array, object, number, bool). Everything downstream,
however, assumes a `str`. This produces two distinct failures depending on the
verb:
| Request | Result |
|---|---|
| `POST /api/v2/variables` with `value` as a JSON array | **201** — silently
stores the Python `repr`, i.e. `['a', 'b']`, which is **not valid JSON** |
| `PATCH /api/v2/variables/{key}` with `value` as a JSON array | **500
Internal Server Error**, real exception masked |
The `POST` case is the more dangerous of the two: it succeeds, so nobody
notices until a DAG calls `Variable.get(key, deserialize_json=True)` and blows
up on a value that can no longer be parsed.
Note the inconsistency: a non-string **`description`** is correctly rejected
with a clean `422`. Only `value` misbehaves.
## Reproduction
```bash
# 1) POST — silently corrupts (HTTP 201)
curl -X POST "$HOST/api/v2/variables" \
-H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
-d '{"key":"my_var","value":["a","b"],"description":"d"}'
# -> 201 {"key":"my_var","value":"['a', 'b']", ...}
# json.loads("['a', 'b']") raises JSONDecodeError
# 2) PATCH — 500
curl -X PATCH "$HOST/api/v2/variables/my_var" \
-H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
-d '{"key":"my_var","value":["a","b"],"description":"d"}'
# -> 500 Internal Server Error
```
For contrast, both verbs behave correctly when `value` is a JSON *string*
(`"[\"a\", \"b\"]"`) — `200`/`201` and the stored value round-trips as valid
JSON.
## Root cause
`airflow-core/src/airflow/api_fastapi/core_api/datamodels/variables.py`:
```python
class VariableBody(StrictBaseModel):
key: str = Field(max_length=ID_LEN)
value: JsonValue = Field(serialization_alias="val") # accepts any JSON
type
description: str | None = Field(default=None)
team_name: str | None = Field(max_length=50, default=None)
```
The two verbs then diverge:
**PATCH** — `update_orm_from_pydantic()` applies the patch straight onto the
ORM instance via `BulkService.apply_patch_with_update_mask()`. `setattr` hits
the `Variable.val` setter in `airflow-core/src/airflow/models/variable.py`:
```python
@val.setter
def set_val(self, value):
if value is not None:
...
self._val = fernet.encrypt(bytes(value, "utf-8")).decode()
```
`bytes(<list>, "utf-8")` raises. Verified in-process:
```python
>>> from airflow.models import Variable
>>> v = Variable(key='k'); v.val = ['a', 'b']
Traceback (most recent call last):
File "airflow/models/variable.py", line 103, in set_val
self._val = fernet.encrypt(bytes(value, "utf-8")).decode()
TypeError: encoding without a string argument
```
**POST** — goes through `Variable.set()`, which coerces instead of
validating:
```python
if serialize_json:
stored_value = json.dumps(value, indent=2)
else:
stored_value = str(value) # list -> "['a', 'b']"
```
`str()` on a list yields Python `repr` with single quotes, so the stored
value is not valid JSON and cannot be recovered by `deserialize_json=True`.
## Middleware masks the real error
The `TypeError` never reaches the logs. `JWTRefreshMiddleware` (a Starlette
`BaseHTTPMiddleware`) surfaces only:
```
File "airflow/api_fastapi/auth/middlewares/refresh_token.py", line 61, in
dispatch
response = await call_next(request)
File "starlette/middleware/base.py", line 169, in call_next
raise RuntimeError("No response returned.")
RuntimeError: No response returned.
```
This made the failure substantially harder to diagnose — the traceback names
neither the field nor the underlying `TypeError`. Same masking behaviour as
reported in #68868 (different trigger, `team_name: ""`) and #66889.
## Suggested fix
Tighten the schema so the API rejects non-string values with a `422`,
consistent with `description`:
```python
value: str | None = Field(serialization_alias="val", default=None)
```
If `JsonValue` must be kept for backwards compatibility, add a
`field_validator` that JSON-serialises non-string input (`json.dumps`) rather
than letting `str()` produce a Python `repr`, and guard `Variable.set_val` so
it raises a validation error rather than a bare `TypeError`.
Separately, it would be worth making `JWTRefreshMiddleware` propagate the
underlying exception so these failures are diagnosable from the API server logs.
### Operating System
Debian GNU/Linux 12 (official `apache/airflow` base image, Python 3.11)
### Versions of Apache Airflow Providers
apache-airflow-providers-fab (FAB auth manager enabled)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow 3.2.1, Python 3.11, Kubernetes, Postgres metadata DB, FAB auth
manager. Reproduced both directly against the api-server pod (`localhost:8080`)
and through the ingress, so it is not proxy-related. Also reproduced with two
different users (roles `Admin` and `Op`), so it is not permission-related.
### Anything else?
Occurs every time. Originally hit via the Web UI when saving a variable
whose value was a JSON array; the UI issues 4 retries with backoff, so a single
click produces 4 x 500 in the api-server access log.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
