1fanwang opened a new pull request, #66888:
URL: https://github.com/apache/airflow/pull/66888
Triggering a Dag run with an oversized `conf` payload (and a whole class of
similarly-shaped writes across the API) currently produces a generic `500
Internal Server Error`. The SQL error surfaces deep in SQLAlchemy as `(1406,
"Data too long for column 'conf' at row 1")` on MySQL, the caller has no signal
that payload size was the cause, and every write endpoint that touches a
length-capped column has the same shape today — `Connection.extra`,
`Variable.val`, `XCom.value`, `TaskInstance.note`, HITL fields, and so on.
This adds a single FastAPI exception handler for `sqlalchemy.exc.DataError`
and registers it on both the public REST API and the task-execution API. `Data
too long` / `too large` / `too big` errors map to `413 Content Too Large`;
other DataErrors (out-of-range, numeric overflow) map to `422 Unprocessable
Entity`. The response body carries the original DB error plus an actionable
hint pointing at either reducing the payload or widening the column type on
MySQL.
Every existing and future write endpoint inherits the translation
automatically. Postgres deployments never hit it (`JSONB` has no length cap);
MySQL deployments get a clear 4xx + remediation hint instead of a generic 500.
This replaces #66787, which proposed a config-knob + per-route validator +
new exception class for the same problem. Closing that one in favour of this
minimal, generalised version.
## Reproducer
A real MySQL 8.0 container reproduces the literal `(1406, ...)` DataError,
and the same exception driven through a FastAPI `TestClient` shows the response
transition.
```
$ docker run --rm -d --name mysql-66787 \
-e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=airflow_test \
-p 3309:3306 mysql:8.0
$ python /tmp/66787_dataerror_repro.py
```
The script inserts a 70 KB JSON payload into a `dag_run`-shaped `TEXT conf`
column to surface the real DataError, then drives the same error through
TestClient with and without `_DataErrorHandler` registered.
### Before this PR
```
=== Step 1: real MySQL DataError (oversized conf into TEXT column) ===
exception class: sqlalchemy.exc.DataError
orig: (1406, "Data too long for column 'conf' at row 1")
=== Step 2: drive the same DataError through FastAPI TestClient ===
--- 2a: upstream/main behavior (no DataError handler registered) ---
upstream/main: HTTP 500
body: Internal Server Error
--- 2b: this PR (DataError -> 413 via _DataErrorHandler) ---
ImportError: cannot import name '_DataErrorHandler' from
'airflow.api_fastapi.common.exceptions'
```
### After this PR
```
=== Step 1: real MySQL DataError (oversized conf into TEXT column) ===
exception class: sqlalchemy.exc.DataError
orig: (1406, "Data too long for column 'conf' at row 1")
=== Step 2: drive the same DataError through FastAPI TestClient ===
--- 2a: upstream/main behavior (no DataError handler registered) ---
upstream/main: HTTP 500
body: Internal Server Error
--- 2b: this PR (DataError -> 413 via _DataErrorHandler) ---
this PR : HTTP 413
body: {"detail":{"reason":"Payload exceeded database column
limit","orig_error":"(1406, \"Data too long for column 'conf' at row
1\")","message":"Database rejected the payload. Reduce the field size, or your
operator may widen the column type (e.g. MEDIUMTEXT / LONGTEXT on MySQL)."}}
```
Same DataError, same `orig_error` field; the difference is the HTTP response
the API caller actually sees.
## Tests
`airflow-core/tests/unit/api_fastapi/common/test_exceptions.py::TestDataErrorHandler`
covers:
- five parametrised dialect-error shapes — MySQL 1406 (`Data too long`),
Postgres `value too long for type ...`, SQLite `string or blob too big`, MySQL
1264 (`Out of range`), Postgres `numeric field overflow` — asserting 413 for
the "too large" family and 422 for the "out of range" family, the original DB
error round-trips in the `orig_error` field, and the `MEDIUMTEXT` hint is in
the message;
- an end-to-end dispatch test that registers `ERROR_HANDLERS` on a `FastAPI`
app, raises `DataError` from a route, and asserts the client receives the 413
with the structured body.
Run on this branch:
```
$ uv run --project airflow-core pytest
airflow-core/tests/unit/api_fastapi/common/test_exceptions.py -v
...
TestDataErrorHandler::test_dataerror_translates_to_actionable_http_response[mysql-1406-data-too-long]
PASSED
TestDataErrorHandler::test_dataerror_translates_to_actionable_http_response[postgres-value-too-long]
PASSED
TestDataErrorHandler::test_dataerror_translates_to_actionable_http_response[sqlite-blob-too-big]
PASSED
TestDataErrorHandler::test_dataerror_translates_to_actionable_http_response[mysql-1264-out-of-range]
PASSED
TestDataErrorHandler::test_dataerror_translates_to_actionable_http_response[postgres-numeric-field-overflow]
PASSED
TestDataErrorHandler::test_dataerror_dispatched_through_fastapi_app PASSED
... 16 passed, 6 skipped
```
On `main` (handler not present) the same suite errors out at import-time
with `ImportError: cannot import name '_DataErrorHandler'`.
## Scope
Strictly `DataError`. `IntegrityError` translation (unique violations are
already handled; FK / NOT NULL violations aren't) is intentionally left for a
follow-up so this stays small and the failure-class is one specific shape — the
database telling the API server "this value didn't fit", which is always a
client-input problem.
Closes #66779.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.7)
Generated-by: Claude Code (Opus 4.7) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
