iharshlalakiya commented on issue #73248: URL: https://github.com/apache/airflow/issues/73248#issuecomment-5701444611
Confirmed — the failing step is the single `ALTER TABLE xcom ALTER COLUMN value TYPE JSONB` in migration 0049. It's one atomic statement, so one unconvertible row aborts the whole migration. One correction: the 268435455-byte cap is on the *elements* of a jsonb array/object, not on the value as a whole. A top-level jsonb string isn't subject to it (the 1 GB varlena limit applies instead), and jsonb can be larger than the text it came from, so a big array of small numbers can be under 268435455 as `bytea` and still fail. `octet_length(value) > 268435455` finds candidates but is neither necessary nor sufficient. Proposed fix — the archive-and-continue option from the description: - Move unconvertible values into the existing `_xcom_archive` table that 0049 already creates for the pickled rows, so no new table is needed, and let the migration proceed. - Decide per row by trying the actual cast in a `pg_temp` function rather than filtering on size, for the reason above. A cheap `octet_length` screen keeps that probe off rows that can't be affected. - Print the `dag_id`/`task_id`/`run_id`/`key` of everything archived, so the list is available either way. - PostgreSQL only — MySQL and SQLite use generic JSON and have no equivalent cap. Does this look like the right direction, particularly amending a released migration in place? Open to a different approach if maintainers prefer fail-early instead. Happy to take this on, could it be assigned to me? Also: this is `airflow-core/src/airflow/migrations/`, not Providers — worth recategorising. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
