ephraimbuddy commented on PR #31445:
URL: https://github.com/apache/airflow/pull/31445#issuecomment-1560883071
I have tested the upgrade and downgrade with 17million rows in taskinstance
and dagrun tables, 15million rows in task_map table.
The test was done with airflow 2.6.1.
Upgrade: `airflow db upgrade 1.26s user 0.20s system 10% cpu 14.438 total`
Downgrade: `airflow db downgrade --to-version 2.6.1 0.78s user 0.17s system
5% cpu 15.969 total`
So a total of 1.26s for upgrade and 0.78s for downgrade
Below is the full log of the test:
```bash
(env) ➜ airflow time airflow db upgrade
DB: postgresql+psycopg2://ephraimbuddy:***@localhost/airflowdb
Performing upgrade with database
postgresql+psycopg2://ephraimbuddy:***@localhost/airflowdb
[2023-05-24T11:21:28.283+0100] {migration.py:207} INFO - Context impl
PostgresqlImpl.
[2023-05-24T11:21:28.283+0100] {migration.py:210} INFO - Will assume
transactional DDL.
[2023-05-24T11:21:28.287+0100] {db.py:1589} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 98ae134e6fff ->
c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Revision ID: c804e5c76e3e
Revises: 98ae134e6fff
Create Date: 2023-05-19 23:30:57.368617
WARNI
[unusual_prefix_1a46e8f1a3ec13e22b7f06b6da27649e667e980d_example_local_kubernetes_executor]
Could not import DAGs in example_local_kubernetes_executor.py
Traceback (most recent call last):
File
"/Users/ephraimbuddy/airflow/env/lib/python3.9/site-packages/airflow/example_dags/example_local_kubernetes_executor.py",
line 37, in <module>
from kubernetes.client import models as k8s
ModuleNotFoundError: No module named 'kubernetes'
WARNI
[unusual_prefix_1a46e8f1a3ec13e22b7f06b6da27649e667e980d_example_local_kubernetes_executor]
Install Kubernetes dependencies with: pip install
apache-airflow[cncf.kubernetes]
WARNI [airflow.task.operators] Dependency <Task(BashOperator):
create_entry_group>, delete_entry_group already registered for DAG:
example_complex
WARNI [airflow.task.operators] Dependency <Task(BashOperator):
delete_entry_group>, create_entry_group already registered for DAG:
example_complex
WARNI [airflow.task.operators] Dependency <Task(BashOperator):
create_entry_gcs>, delete_entry already registered for DAG: example_complex
WARNI [airflow.task.operators] Dependency <Task(BashOperator):
delete_entry>, create_entry_gcs already registered for DAG: example_complex
WARNI [airflow.task.operators] Dependency <Task(BashOperator): create_tag>,
delete_tag already registered for DAG: example_complex
WARNI [airflow.task.operators] Dependency <Task(BashOperator): delete_tag>,
create_tag already registered for DAG: example_complex
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
print_the_context>, log_sql_query already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
log_sql_query>, print_the_context already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
print_the_context>, log_sql_query already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
log_sql_query>, print_the_context already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
print_the_context>, log_sql_query already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
log_sql_query>, print_the_context already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
print_the_context>, log_sql_query already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
log_sql_query>, print_the_context already registered for DAG:
example_python_operator
WARNI [airflow.task.operators] Dependency <Task(_PythonDecoratedOperator):
prepare_email>, send_email already registered for DAG: example_dag_decorator
WARNI [airflow.task.operators] Dependency <Task(EmailOperator): send_email>,
prepare_email already registered for DAG: example_dag_decorator
WARNI
[unusual_prefix_40955236974b30b56bf66303de6e0381cb92c3e4_example_kubernetes_executor]
The example_kubernetes_executor example DAG requires the kubernetes provider.
Please install it with: pip install apache-airflow[cncf.kubernetes]
WARNI [unusual_prefix_ef50229d1e6ad64f48f32a8c7d705159d326694c_workday]
Could not import pandas. Holidays will not be considered.
Upgrades done
airflow db upgrade 1.26s user 0.20s system 10% cpu 14.438 total
(env) ➜ airflow time airflow db downgrade --to-version 2.6.1
Performing downgrade with database
postgresql+psycopg2://ephraimbuddy:***@localhost/airflowdb
Warning: About to reverse schema migrations for the airflow metastore.
Please ensure you have backed up your database before any upgrade or downgrade
operation. Proceed? (y/n)
y
[2023-05-24T11:22:08.464+0100] {db.py:1667} INFO - Attempting downgrade to
revision 98ae134e6fff
/Users/ephraimbuddy/airflow/env/lib/python3.9/site-packages/airflow/utils/db.py:1769
RemovedIn20Warning: [31mDeprecated API features detected! These feature(s) are
not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior
to updating applications, ensure requirements files are pinned to
"sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all
deprecation warnings. Set environment variable
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-05-24T11:22:08.468+0100] {db.py:1678} INFO - Applying downgrade
migrations.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running downgrade c804e5c76e3e ->
98ae134e6fff, Add ``onupdate`` cascade to ``task_map`` table
Revision ID: c804e5c76e3e
Revises: 98ae134e6fff
Create Date: 2023-05-19 23:30:57.368617
Downgrade complete
airflow db downgrade --to-version 2.6.1 0.78s user 0.17s system 5% cpu
15.969 total
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
