ChandelAnushka opened a new issue, #27836:
URL: https://github.com/apache/airflow/issues/27836
### Apache Airflow version
2.4.3
### What happened
My production deployment had Airflow v2.1.0. It already had Dags, Task
Instances, etc., in the tables. When upgrading to Airflow v2.4.3, the execution
of airflow db upgrade resulted in the failure: sqlalchemy.exc.ProgrammingError:
(psycopg2.ProgrammingError) column "run_id" referenced in foreign key
constraint does not exist
The logs/traceback:
```
[2022-11-22 04:10:31,786] {migration.py:201} INFO - Context impl
PostgresqlImpl.
[2022-11-22 04:10:31,787] {migration.py:208} INFO - Will assume
transactional DDL.
[2022-11-22 04:10:31,789] {migration.py:201} INFO - Context impl
PostgresqlImpl.
[2022-11-22 04:10:31,790] {migration.py:208} INFO - Will assume
transactional DDL.
Traceback (most recent call last):
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
1901, in _execute_context
cursor, statement, parameters, context
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line
736, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: column "run_id" referenced in foreign key
constraint does not exist
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/a/bin/airflow", line 8, in <module>
sys.exit(main())
File "/a/lib/python3.7/site-packages/airflow/__main__.py", line 39, in main
args.func(args)
File "/a/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52,
in command
return func(*args, **kwargs)
File "/a/lib/python3.7/site-packages/airflow/utils/cli.py", line 103, in
wrapper
return f(*args, **kwargs)
File "/a/lib/python3.7/site-packages/airflow/cli/commands/db_command.py",
line 84, in upgradedb
db.upgradedb(to_revision=to_revision, from_revision=from_revision,
show_sql_only=args.show_sql_only)
File "/a/lib/python3.7/site-packages/airflow/utils/session.py", line 75,
in wrapper
return func(*args, session=session, **kwargs)
File "/a/lib/python3.7/site-packages/airflow/utils/db.py", line 1560, in
upgradedb
initdb(session=session, load_connections=False)
File "/a/lib/python3.7/site-packages/airflow/utils/session.py", line 72,
in wrapper
return func(*args, **kwargs)
File "/a/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in
initdb
_create_db_from_orm(session=session)
File "/a/lib/python3.7/site-packages/airflow/utils/db.py", line 667, in
_create_db_from_orm
Base.metadata.create_all(settings.engine)
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4931,
in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
3228, in _run_ddl_visitor
conn._run_ddl_visitor(visitorcallable, element, **kwargs)
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
2211, in _run_ddl_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line
524, in traverse_single
return meth(obj, **kw)
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 855, in
visit_metadata
_is_metadata_operation=True,
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line
524, in traverse_single
return meth(obj, **kw)
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 900, in
visit_table
include_foreign_key_constraints, # noqa
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/a/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 81, in
_execute_on_connection
self, multiparams, params, execution_options
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
1478, in _execute_ddl
compiled,
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
1944, in _execute_context
e, statement, parameters, cursor, context
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
2125, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/a/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 210,
in raise_
raise exception
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line
1901, in _execute_context
cursor, statement, parameters, context
File "/a/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line
736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "run_id"
referenced in foreign key constraint does not exist
[SQL:
CREATE TABLE task_map (
dag_id VARCHAR(250) NOT NULL,
task_id VARCHAR(250) NOT NULL,
run_id VARCHAR(250) NOT NULL,
map_index INTEGER NOT NULL,
length INTEGER NOT NULL,
keys JSON,
CONSTRAINT task_map_pkey PRIMARY KEY (dag_id, task_id, run_id,
map_index),
CONSTRAINT ck_task_map_task_map_length_not_negative CHECK (length >=
0),
CONSTRAINT task_map_task_instance_fkey FOREIGN KEY(dag_id, task_id,
run_id, map_index) REFERENCES task_instance (dag_id, task_id, run_id,
map_index) ON DELETE CASCADE
)
```
### What you think should happen instead
The database upgrade should have succeeded.
### How to reproduce
Install Airflow 2.1.0 and initialize airflow db.
Create and execute some dags so that Dag table gets populated.
Upgrade airflow to 2.4.3 and execute airflow db upgrade.
### Operating System
Mac OS Monterey
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon 6.0.0
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-ftp 3.1.0
apache-airflow-providers-imap 3.0.0
apache-airflow-providers-papermill 3.0.0
apache-airflow-providers-postgres 5.2.0
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-sqlite 3.2.1
apache-airflow-providers-ssh 3.2.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
