gontcharovd opened a new issue #14320: URL: https://github.com/apache/airflow/issues/14320
**Apache Airflow version**: 2.0.1 **Environment**: <details><summary>Python requirements.txt</summary> alembic==1.5.4 apache-airflow==2.0.1 apache-airflow-providers-ftp==1.0.1 apache-airflow-providers-http==1.1.0 apache-airflow-providers-imap==1.0.1 apache-airflow-providers-sqlite==1.0.1 apispec==3.3.2 argcomplete==1.12.2 attrs==20.3.0 Babel==2.9.0 cached-property==1.5.2 cattrs==1.2.0 certifi==2020.12.5 cffi==1.14.5 chardet==4.0.0 click==7.1.2 clickclick==20.10.2 colorama==0.4.4 colorlog==4.7.2 commonmark==0.9.1 connexion==2.7.0 croniter==0.3.37 cryptography==3.4.6 defusedxml==0.6.0 dill==0.3.3 dnspython==2.1.0 docutils==0.16 email-validator==1.1.2 Flask==1.1.2 Flask-AppBuilder==3.1.1 Flask-Babel==1.0.0 Flask-Caching==1.9.0 Flask-JWT-Extended==3.25.1 Flask-Login==0.4.1 Flask-OpenID==1.2.5 Flask-SQLAlchemy==2.4.4 Flask-WTF==0.14.3 graphviz==0.16 gunicorn==19.10.0 idna==2.10 importlib-metadata==1.7.0 importlib-resources==1.5.0 inflection==0.5.1 iniconfig==1.1.1 iso8601==0.1.14 itsdangerous==1.1.0 Jinja2==2.11.3 jsonschema==3.2.0 lazy-object-proxy==1.5.2 lockfile==0.12.2 Mako==1.1.4 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.10.0 marshmallow-enum==1.5.1 marshmallow-oneofschema==2.1.0 marshmallow-sqlalchemy==0.23.1 natsort==7.1.1 numpy==1.20.1 openapi-spec-validator==0.2.9 packaging==20.9 pandas==1.2.2 pendulum==2.1.2 pluggy==0.13.1 prison==0.1.3 psutil==5.8.0 py==1.10.0 pycparser==2.20 Pygments==2.8.0 PyJWT==1.7.1 pyparsing==2.4.7 pyrsistent==0.17.3 pytest==6.2.2 python-daemon==2.2.4 python-dateutil==2.8.1 python-editor==1.0.4 python-nvd3==0.15.0 python-slugify==4.0.1 python3-openid==3.2.0 pytz==2021.1 pytzdata==2020.1 PyYAML==5.4.1 requests==2.25.1 rich==9.2.0 setproctitle==1.2.2 six==1.15.0 SQLAlchemy==1.3.23 SQLAlchemy-JSONField==1.0.0 SQLAlchemy-Utils==0.36.8 swagger-ui-bundle==0.0.8 tabulate==0.8.8 tenacity==6.2.0 termcolor==1.1.0 text-unidecode==1.3 toml==0.10.2 typing-extensions==3.7.4.3 unicodecsv==0.14.1 urllib3==1.26.3 Werkzeug==1.0.1 WTForms==2.3.3 zipp==3.4.0 </details> - **Cloud provider or hardware configuration**: local computer - **OS** (e.g. from /etc/os-release): NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://www.archlinux.org/"; DOCUMENTATION_URL="https://wiki.archlinux.org/"; SUPPORT_URL="https://bbs.archlinux.org/"; BUG_REPORT_URL="https://bugs.archlinux.org/"; LOGO=archlinux - **Kernel** (e.g. `uname -a`): Linux ws 5.11.0-arch1-1 #1 SMP PREEMPT Mon, 15 Feb 2021 23:07:45 +0000 x86_64 GNU/Linux - **Install tools**: / - **Others**: / **What happened**: A simple Airflow task involving the BashOperator is tested using pytest. Let's say the task context is required to, for example, render templated arguments. This means that besides the task two additional things are required: 1. A DAG instance: will be supplied through a pytest fixture 2. The metastore: will be instantiated as an SQLite during the test Even though the task and the test succeed, an `sqlalchemy.orm.exc.NoResultFound` is raised. **What you expected to happen**: I expect the task to run without raising any exceptions because: 1. The task run succeeds as can be read from the `task_instance` table in the SQLite metastore (see below). 2. The test passes 3. The SQLite metastore is fresh (initialized at the start of the test) **How to reproduce it**: The code below contains a minimal example that reproduces the exception on a first run only. To run the task again the `task_id` must be changed. Alternatively the SQLite metastore must be reset or deleted. The objective of the (arbitrary) test is to assert whether the BashOperator succeeds at writing "hello" into a file `hello.txt`. The test relies on this `test_dag` fixture defined in a `conftest.py` file in the same directory as the test file: ```python import datetime import pytest from airflow.models import DAG @pytest.fixture def test_dag(): args = {'owner': 'airflow', 'start_date': datetime.datetime(2020, 1, 1)} return DAG('test_dag', default_args=args, schedule_interval='@daily') ``` This is the minimal test itself written in a file `test_bash_operator.py`: ```python import os import subprocess from airflow.operators.bash import BashOperator from pathlib import Path from sqlalchemy.orm.exc import NoResultFound def test_bash(tmp_path: Path, test_dag, monkeypatch): my_file = tmp_path / 'hello.txt' # initialize the Airflow sqlite metastore in the current folder metastore_path = os.path.join(os.path.dirname(__file__), 'airflow.db') uri = f'sqlite:///{metastore_path}' monkeypatch.setenv('AIRFLOW__CORE__SQL_ALCHEMY_CONN', uri) monkeypatch.setenv('AIRFLOW__CORE__LOAD_EXAMPLES', 'False') monkeypatch.setenv('AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS', 'False') init_metastore = subprocess.Popen(['airflow', 'db', 'init']) init_metastore.wait() print('Created Airflow metastore') task = BashOperator( task_id='test_id', bash_command=f'echo hello > {my_file}', dag=test_dag ) try: print('Running task...') task.run( start_date=test_dag.default_args['start_date'], end_date=test_dag.default_args['start_date'] ) except NoResultFound: print('Caught expected `sqlalchemy.orm.exc.NoResultFound` exception') with open(my_file, 'r') as f: assert f.read().replace('\n', '') == 'hello' ``` Note that the test initializes a SQLite metastore in the same folder. The test's stdout shows that the `sqlalchemy.orm.exc.NoResultFound` is raised and caught: ```sqlite:////home/denis/code/airflow-test/airflow.db DB: sqlite:////home/denis/code/airflow-test/airflow.db [2021-02-19 16:55:17,809] {db.py:674} INFO - Creating tables [2021-02-19 16:55:19,181] {manager.py:727} WARNING - No user yet created, use flask fab command to do it. [2021-02-19 16:55:21,527] {manager.py:727} WARNING - No user yet created, use flask fab command to do it. [2021-02-19 16:55:22,701] {dagbag.py:448} INFO - Filling up the DagBag from /home/denis/airflow/dags Initialization done Created Airflow metastore Running task... Caught expected `sqlalchemy.orm.exc.NoResultFound` exception ``` The SQLite metastore shows that the task run was succesful: ```shell ~ $ airflow db shell DB: sqlite:////home/denis/code/airflow-test/airflow.db [2021-02-19 17:23:10,015] {process_utils.py:154} INFO - Executing cmd: sqlite3 /home/denis/code/airflow-test/airflow.db SQLite version 3.34.1 2021-01-20 14:10:07 Enter ".help" for usage hints. sqlite> select * from task_instance ; test_id|test_dag|2020-01-01 00:00:00.000000|2021-02-19 15:55:22.980135|2021-02-19 15:55:23.033481|0.053346|success|1|ws.localdomain|denis||default_pool|default|1|BashOperator||956921|0|}.|1|| ``` The test itself passes, meaning that the BashOperator successfully creates the hello.txt file. So if everything seems to work as intended, why is the `sqlalchemy.orm.exc.NoResultFound` raised? *** ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
