potiuk opened a new issue, #44377:
URL: https://github.com/apache/airflow/issues/44377
Sometimes our test fail when you develop airlfow inlocal venv with "missing"
or "wrong" database objects - such as tables, sequences.
For local venv (which almost exclusively uses `sqlite` until we implement
`test-containers` produces failures related to sqlalchemy . Specifically when
running individual tests using commands like
`pytest
tests/api_fastapi/core_api/routes/public/test_dag_run.py::TestTriggerDagRun::test_raises_validation_error_for_invalid_params`
It happens to one of the fastapi tests
They just randomly happen for different tables and the error is something
like this:
```
E sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such
table: dag_run_note
E [SQL: SELECT dag_run_note.user_id AS dag_run_note_user_id,
dag_run_note.dag_run_id AS dag_run_note_dag_run_id, dag_run_note.content AS
dag_run_note_content, dag_run_note.created_at AS dag_run_note_created_at,
dag_run_note.updated_at AS dag_run_note_updated_at
E FROM dag_run_note
E WHERE dag_run_note.dag_run_id = ?]
E [parameters: (1,)]
E (Background on this error at: https://sqlalche.me/e/14/e3q8)
```
if this happens, you need to likely reset your database
A lot of tests in Airflow rely on database being available (so called
"db_tests" and they expect it to be in the "right" state - i.e. tables created
(edited)
This will happen automatically if you run tests for the first time - and you
can also force this is initialization with --with-db-init flag, We have
actually quite a number of those airflow-specific pytest flags defined (they
are printed with pytest --help:
```
airflow:
--with-db-init Forces database initialization before tests
--integration=INTEGRATIONS
only run tests matching integration specified:
[cassandra,kerberos,mongo,celery,statsd,trino].
--keep-env-variables do not clear environment variables that might have
side effect while running tests
--skip-db-tests skip tests that require database
--run-db-tests-only only run tests requiring database
--backend=BACKEND only run tests matching the backend:
[sqlite,postgres,mysql].
--system run system tests
--include-long-running
Includes long running tests (marked with
long_running marker). They are skipped by default.
--include-quarantined
Includes quarantined tests (marked with quarantined
marker). They are skipped by default.
--trace-sql=COLUMNS Trace SQL statements. As an argument, you must
specify the columns to be displayed as a comma-separated list. Supported
values: [fnum,time,trace,sql,parameters,count]
--no-db-cleanup Disable DB clear before each test module.
--disable-forbidden-warnings
Disable raising an error if forbidden warnings
detected.
--disable-capture-warnings
Disable internal capture warnings.
--warning-output-path=PATH
Path for resulting captured warnings. Absolute or
relative to the `tests` directory. If not provided or environment variable
`CAPTURE_WARNINGS_OUTPUT` not set then 'warnings.txt' will be used.
```
By default the database is stored in ~/airflow/ folder - it's an sqlite
database.
And if this happens you have to delete it manually while developing -
especially after our recent agreement that we might squash some migrations
while developing the DB - because it might mean that a database created at some
point in time during your development, might not be "upgrade'able" any more -
i.e. neither --with-db-init pytest flag nor airflow db reset will work (both
will crash - and --with-db-init is I think crashing silently then)
So sometimes you might have to manually wipe-out the local database and
start from scratch. This is what `breeze down` command is doing for you - it
deletes all the files that store created database - but breeze does it not only
for `sqlite` but also for `postgres` and `mysql`.
The `initialize_airflow_tests` fixture in `tests/common/ptyest_plugin.py`
performs this initialization automatically when you first time run tests, buet
then it does not di maintenance automatically and does not handle all cases -
especially when database changes got reverted or squashed while developing the
db.
We could likely improve the code in that fixture to make it more resilient
and "self-healing" the sqlite database, or maybe at the very least have better
error messages when it happens and instruct the contributors what to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
