[jira] [Commented] (AIRFLOW-1452) "airflow initdb" stuck forever on upgrade
[
https://issues.apache.org/jira/browse/AIRFLOW-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123837#comment-16123837
]
ASF subversion and git services commented on AIRFLOW-1452:
--
Commit 0d0cc62f49525166bc877606affa5a623ba52c4d in incubator-airflow's branch
refs/heads/master from k.privezentsev
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0d0cc62 ]
[AIRFLOW-1452] workaround lock on method
Workaround lock on method "has_table" in case
mssql is used
as storage engine.
Closes #2514 from patsak/master
> "airflow initdb" stuck forever on upgrade
> -
>
> Key: AIRFLOW-1452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1452
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
>Reporter: Pavel Martynov
> Attachments: docker-compose.yml, Dockerfile, run-initdb.sh
>
>
> I install airflow from the current master branch
> (426b6a65f6ec142449893e36fcd677941bdad879 when I write this issue) and run
> "airflow initdb" against MS SQL and it stuck forever with that output:
> {noformat}
> [2017-07-25 07:30:12,458] {db.py:307} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> {noformat}
> I reproduce this problem with docker-compose, see files in attachment.
> Also, I try this on 1.8.2rc2 and it works fine, looks like problem in
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration.
> Some locks occurred, I "killed lock" in MS SQL and got exception:
> {noformat}
> sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08S01', '[08S01] [Microsoft][ODBC
> Driver 13 for SQL Server]TCP Provider: Error code 0x2746 (10054)
> (SQLExecDirectW)') [SQL: u"UPDATE alembic_version SET
> version_num='cc1e65623dc7' WHERE alembic_version.version_num =
> '127d2bf2dfa7'"]
> {noformat}
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration was added for
> [AIRFLOW-1366].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1452) "airflow initdb" stuck forever on upgrade
[
https://issues.apache.org/jira/browse/AIRFLOW-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106813#comment-16106813
]
Pavel Martynov commented on AIRFLOW-1452:
-
[~allisonwang] I understand that this migration may take a lot of time for
existing Airflow DB instance, but I run {{airflow initdb}} for initialization
of new DB instance without any existing task_instance rows. So initialization
in my case is not slow but stuck forever.
Also, please pay attention, I talking about MS SQL, not MySQL. I ran into this
issue initially with MS SQL 2016 running on Windows Server, but reproduced this
issue with MS SQL 2017rc1 on Linux with Docker, please look at files in
attachment. To reproduce an issue you can just run {{docker-compose up}}.
> "airflow initdb" stuck forever on upgrade
> -
>
> Key: AIRFLOW-1452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1452
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
>Reporter: Pavel Martynov
> Attachments: docker-compose.yml, Dockerfile, run-initdb.sh
>
>
> I install airflow from the current master branch
> (426b6a65f6ec142449893e36fcd677941bdad879 when I write this issue) and run
> "airflow initdb" against MS SQL and it stuck forever with that output:
> {noformat}
> [2017-07-25 07:30:12,458] {db.py:307} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> {noformat}
> I reproduce this problem with docker-compose, see files in attachment.
> Also, I try this on 1.8.2rc2 and it works fine, looks like problem in
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration.
> Some locks occurred, I "killed lock" in MS SQL and got exception:
> {noformat}
> sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08S01', '[08S01] [Microsoft][ODBC
> Driver 13 for SQL Server]TCP Provider: Error code 0x2746 (10054)
> (SQLExecDirectW)') [SQL: u"UPDATE alembic_version SET
> version_num='cc1e65623dc7' WHERE alembic_version.version_num =
> '127d2bf2dfa7'"]
> {noformat}
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration was added for
> [AIRFLOW-1366].
--
This message was sent by Atlassian JIRA
[jira] [Commented] (AIRFLOW-1452) "airflow initdb" stuck forever on upgrade
[
https://issues.apache.org/jira/browse/AIRFLOW-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106261#comment-16106261
]
Allison Wang commented on AIRFLOW-1452:
---
Hi Pavel. This migration indeed takes a long time, assuming you do not have
mysql database connection anywhere else. We did a profiling for the migration
on 1M rows and it takes about an hour. This is because the new column populates
its value from existing rows and does an UPDATE query for each task_instance
row.
If the process is taking more than an hour, it means the database still holds
locks on some task_instance table rows. Please make sure to disconnect
everything before upgrading.
We are aware this slow migration and will address it ASAP. In the mean while, I
highly suggest using an older version of Airflow other than the master branch.
There is a progress going on to refactor and improve airflow logging on the
master branch.
Please let us know if you have more questions or concerns.
> "airflow initdb" stuck forever on upgrade
> -
>
> Key: AIRFLOW-1452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1452
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
>Reporter: Pavel Martynov
> Attachments: docker-compose.yml, Dockerfile, run-initdb.sh
>
>
> I install airflow from the current master branch
> (426b6a65f6ec142449893e36fcd677941bdad879 when I write this issue) and run
> "airflow initdb" against MS SQL and it stuck forever with that output:
> {noformat}
> [2017-07-25 07:30:12,458] {db.py:307} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> {noformat}
> I reproduce this problem with docker-compose, see files in attachment.
> Also, I try this on 1.8.2rc2 and it works fine, looks like problem in
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration.
> Some locks occurred, I "killed lock" in MS SQL and got exception:
> {noformat}
> sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08S01', '[08S01] [Microsoft][ODBC
> Driver 13 for SQL Server]TCP Provider: Error code 0x2746 (10054)
> (SQLExecDirectW)') [SQL: u"UPDATE alembic_version SET
>
[jira] [Commented] (AIRFLOW-1452) "airflow initdb" stuck forever on upgrade
[
https://issues.apache.org/jira/browse/AIRFLOW-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099808#comment-16099808
]
Pavel Martynov commented on AIRFLOW-1452:
-
[~allisonwang] can you look at this, please?
> "airflow initdb" stuck forever on upgrade
> -
>
> Key: AIRFLOW-1452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1452
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
>Reporter: Pavel Martynov
> Attachments: docker-compose.yml, Dockerfile, run-initdb.sh
>
>
> I install airflow from the current master branch
> (426b6a65f6ec142449893e36fcd677941bdad879 when I write this issue) and run
> "airflow initdb" against MS SQL and it stuck forever with that output:
> {noformat}
> [2017-07-25 07:30:12,458] {db.py:307} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 ->
> 1507a7289a2f, create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 ->
> 52d714495f0, job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 ->
> 502898887f84, Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> {noformat}
> I reproduce this problem with docker-compose, see files in attachment.
> Also, I try this on 1.8.2rc2 and it works fine, looks like problem in
> cc1e65623dc7_add_max_tries_column_to_task_instance.py migration.
> Some locks occurred, I "killed lock" in MS SQL and got exception:
> {noformat}
> sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08S01', '[08S01] [Microsoft][ODBC
> Driver 13 for SQL Server]TCP Provider: Error code 0x2746 (10054)
> (SQLExecDirectW)') [SQL: u"UPDATE alembic_version SET
> version_num='cc1e65623dc7' WHERE alembic_version.version_num =
> '127d2bf2dfa7'"]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
