Jarek Potiuk created AIRFLOW-6265:
-------------------------------------
Summary: Migration might behave wrongly when DAGS parsed use DB
Key: AIRFLOW-6265
URL: https://issues.apache.org/jira/browse/AIRFLOW-6265
Project: Apache Airflow
Issue Type: Bug
Components: models
Affects Versions: 1.10.6, 1.10.5, 1.10.4, 1.10.3, 1.10.2
Reporter: Jarek Potiuk
When running migratedb there might be a problem displayed during migration - in
case some of the DAGs of the user use the DB in __init__ (for example reading
connection table) they might fail mid-migration with errors for example the one
below.
The root cause is
[https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py#L70]
- DagBag object creation which parses all the DAGs.
It it is a minor annoyance - some of the TaskInstances can be not migrated
properly with the max_tries not copied from task to task_instance.
airflow | ========= ABOUT TO INIT DB ===========
...
airflow | DB:
[postgresql+psycopg2://airflow]
:***@postgres:5432/airflow
airflow | [2019-12-17 18:17:06,471] \{db.py:350} INFO - Creating tables
airflow | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
airflow | INFO [alembic.runtime.migration] Will assume transactional DDL.
airflow | INFO [alembic.runtime.migration] Running upgrade ->
e3a246e0dc1, current schema
airflow | INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
1507a7289a2f, create is_encrypted
airflow | INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
13eb55f81627, maintain history for compatibility with earlier migrations
...
airflow_db | 2019-12-17 18:17:07.001 UTC [55] ERROR: relation "connection"
does not exist at character 481
airflow_db | 2019-12-17 18:17:07.001 UTC [55] STATEMENT: SELECT
connection.password AS connection_password, connection.extra AS
connection_extra, connection.id AS connection_id, connection.conn_id AS
connection_conn_id, connection.conn_type AS connection_conn_type,
connection.host AS connection_host, connection.schema AS connection_schema,
connection.login AS connection_login, connection.port AS connection_port,
connection.is_encrypted AS connection_is_encrypted,
connection.is_extra_encrypted AS connection_is_extra_encrypted
airflow_db | FROM connection
airflow_db | WHERE connection.conn_id = 'redshift_test_connection'
airflow | ERROR [airflow.models.DagBag] Failed to import:
/airflow/repo/dags/papermill_test/test_papermill_connection_injection/papermill_dag.py
airflow | Traceback (most recent call last):
airflow | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236,
in _execute_context
airflow | cursor, statement, parameters, context
airflow | File
"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
536, in do_execute
airflow | cursor.execute(statement, parameters)
airflow | psycopg2.errors.UndefinedTable: relation "connection" does not
exist
airflow | LINE 2: FROM connection
...
airflow | INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 ->
bdaa763e6c56, Make xcom value column a large binary
airflow | INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 ->
947454bf1dff, add ti job_id index
airflow | INFO [alembic.runtime.migration] Running upgrade 947454bf1dff ->
d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text
types)
airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
0e2a74e0fc9f, Add time zone awareness
airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
33ae817a1ff4, kubernetes_resource_checkpointing
airflow | INFO [alembic.runtime.migration] Running upgrade 33ae817a1ff4 ->
27c6a30d7c24, kubernetes_resource_checkpointing
...
========= SLEEPING ===========
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
