[
https://issues.apache.org/jira/browse/AIRFLOW-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk updated AIRFLOW-6265:
----------------------------------
Affects Version/s: 2.0.0
> Migration might behave wrongly when DAGS parsed use DB
> ------------------------------------------------------
>
> Key: AIRFLOW-6265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6265
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
> Affects Versions: 1.10.2, 1.10.3, 1.10.4, 2.0.0, 1.10.5, 1.10.6
> Reporter: Jarek Potiuk
> Priority: Minor
>
> When running migratedb there might be a problem displayed during migration -
> in case some of the DAGs of the user use the DB in __init__ (for example
> reading connection table) they might fail mid-migration with errors for
> example the one below.
>
> The root cause is
> [https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py#L70]
> - DagBag object creation which parses all the DAGs.
>
> It it is a minor annoyance - some of the TaskInstances can be not migrated
> properly with the max_tries not copied from task to task_instance.
> airflow | ========= ABOUT TO INIT DB ===========
> ...
> airflow | DB:
> [postgresql+psycopg2://airflow]
> :***@postgres:5432/airflow
> airflow | [2019-12-17 18:17:06,471] \{db.py:350} INFO - Creating tables
> airflow | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> airflow | INFO [alembic.runtime.migration] Will assume transactional DDL.
> airflow | INFO [alembic.runtime.migration] Running upgrade ->
> e3a246e0dc1, current schema
> airflow | INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1
> -> 1507a7289a2f, create is_encrypted
> airflow | INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f
> -> 13eb55f81627, maintain history for compatibility with earlier migrations
> ...
> airflow_db | 2019-12-17 18:17:07.001 UTC [55] ERROR: relation "connection"
> does not exist at character 481
> airflow_db | 2019-12-17 18:17:07.001 UTC [55] STATEMENT: SELECT
> connection.password AS connection_password, connection.extra AS
> connection_extra, connection.id AS connection_id, connection.conn_id AS
> connection_conn_id, connection.conn_type AS connection_conn_type,
> connection.host AS connection_host, connection.schema AS connection_schema,
> connection.login AS connection_login, connection.port AS connection_port,
> connection.is_encrypted AS connection_is_encrypted,
> connection.is_extra_encrypted AS connection_is_extra_encrypted
> airflow_db | FROM connection
> airflow_db | WHERE connection.conn_id = 'redshift_test_connection'
> airflow | ERROR [airflow.models.DagBag] Failed to import:
> /airflow/repo/dags/papermill_test/test_papermill_connection_injection/papermill_dag.py
> airflow | Traceback (most recent call last):
> airflow | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line
> 1236, in _execute_context
> airflow | cursor, statement, parameters, context
> airflow | File
> "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line
> 536, in do_execute
> airflow | cursor.execute(statement, parameters)
> airflow | psycopg2.errors.UndefinedTable: relation "connection" does not
> exist
> airflow | LINE 2: FROM connection
> ...
> airflow | INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7
> -> bdaa763e6c56, Make xcom value column a large binary
> airflow | INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56
> -> 947454bf1dff, add ti job_id index
> airflow | INFO [alembic.runtime.migration] Running upgrade 947454bf1dff
> -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs'
> text types)
> airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61
> -> 0e2a74e0fc9f, Add time zone awareness
> airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61
> -> 33ae817a1ff4, kubernetes_resource_checkpointing
> airflow | INFO [alembic.runtime.migration] Running upgrade 33ae817a1ff4
> -> 27c6a30d7c24, kubernetes_resource_checkpointing
> ...
> ========= SLEEPING ===========
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
