seanmuth opened a new pull request, #69985:
URL: https://github.com/apache/airflow/pull/69985
Closes #69980.
## What's the problem
Migration `0094_3_2_0_replace_deadline_inline_callback_with_fkey` moves
inline deadline callbacks into the `callback` table, whose `data` column is
`ExtendedJSON` — read back at runtime via `BaseSerialization.deserialize`,
which requires every nested dict to be wrapped as `{"__type": "dict", "__var":
{...}}`.
Both upgrade paths built `callback.data` without recursively
extended-serializing the callback `kwargs`:
- **Postgres** (`_upgrade_postgresql`): `json_build_object` wrapped only the
top level, embedding the old callback's `kwargs` jsonb raw.
- **MySQL/SQLite** (`_upgrade_mysql_sqlite`): wrote the callback dict via a
plain `JSON` column with no extended wrapping at all.
So any callback whose `kwargs` contain a nested dict (e.g. metric `tags`) is
stored unreadable. When the scheduler loads it in the deadline-processing loop,
`BaseSerialization.deserialize` recurses into the outer `dict` node, hits the
bare nested object, runs `encoded_var[Encoding.VAR]`, and raises
`KeyError(<Encoding.VAR: '__var'>)` — crashing the scheduler on every loop
(CrashLoopBackOff), before it heartbeats, with no OOM and no other logged
exception.
## What's in this PR
- Both **upgrade** paths now extended-serialize the callback data, matching
`BaseSerialization.serialize`:
- Postgres: a recursive, session-local `pg_temp.encode_extended(jsonb)`
function used inside the existing batch CTE (keeps the one-round-trip design).
- MySQL/SQLite: a `_serialize_extended()` helper applied to the callback
dict.
- Both **downgrade** paths decode `kwargs` back to raw
(`pg_temp.decode_extended` / `_deserialize_extended`), so an upgrade→downgrade
round-trip is preserved. Both decoders are lenient, so they also correctly
downgrade data written by the pre-fix version of this migration.
- **Tests** (`test_0094_deadline_callback_migration.py`): existing
assertions now verify `callback.data` round-trips through the real
`BaseSerialization`; added a nested-`kwargs`/`tags` regression test and an
encode/decode helper round-trip test.
## Already-affected deployments
This hotfix protects deployments that have **not yet** run `0094`.
Deployments already upgraded across `0094` with nested-dict callback kwargs
have latent corrupt `callback.data` rows (a scheduler crash that fires when
such a deadline becomes due); those rows must be repaired out of band — e.g.
re-encode the affected rows:
```sql
-- find affected rows
SELECT count(*) FROM callback c
WHERE jsonb_path_exists(c.data::jsonb, '$.**.__var.* ? (@.type()=="object"
&& !exists(@.__type))');
```
---
🤖 Opened with [Claude Code](https://claude.com/claude-code) (model: Claude
Opus 4.8, `claude-opus-4-8[1m]`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
