payalbisen-svg opened a new issue, #73248:
URL: https://github.com/apache/airflow/issues/73248
### Under which category would you file this issue?
Providers
### Apache Airflow version
upgrading from 2.11.2 to [3.x from Runtime 3.3-7] Database: PostgreSQL
### What happened and how to reproduce it?
On PostgreSQL, xcom.value is bytea in Airflow 2, which allows up to 1 GB per
value. Migration 0049 (remove_pickled_data_from_xcom_table)
converts it to jsonb, which caps a single value at 268,435,455 bytes (~268
MB). XCom values that were valid in Airflow 2 cannot be converted, and the
migration aborts with:
` ProgramLimitExceeded: total size of jsonb array elements exceeds the
maximum of 268435455 bytes`
Alembic stays at 9fc3fc5de720 and every retry fails at the same step.
The upgrade cannot complete until the rows are removed.
Column definition in airflow-core/src/airflow/models/xcom.py: value =
mapped_column(JSON().with_variant(postgresql.JSONB, "postgresql"),
nullable=True)
This is specific to PostgreSQL. Other backends use generic JSON.
**Details:**
- The xcom table was ~30 GB in total. The limit is per value, not per table.
- 55 rows failed, each a task return_value close to the limit.
**How to reproduce:**
On an Airflow 2 PostgreSQL metadata DB, store an XCom value larger than
268,435,455 bytes, then run the Airflow 3 migrations.
**Query to find affected rows before upgrading:**
```
SELECT dag_id, task_id, run_id, key, octet_length(value) AS bytes
FROM xcom
WHERE octet_length(value) > 268435455
ORDER BY bytes DESC;
```
**Workaround:**
Copied the affected rows to a backup table, deleted them from xcom,
and re-ran the migration. It completed.
### What you think should happen instead?
Migration 0049 should move values that cannot fit in jsonb into a quarantine
table (like the existing _airflow_moved__* tables) and continue, or fail early
with a clear list of affected rows. The upgrade docs should state the jsonb
per-value limit for PostgreSQL.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
