mikaeld commented on PR #24588:
URL: https://github.com/apache/airflow/pull/24588#issuecomment-1511653657
@potiuk I've been trying to figure out some of the test failures. However,
I'm not certain how to debug the failing tests. I'm using Breeze because I
could not get those tests to work using a local venv. Maybe I'm just not used
to these kind of tests and not looking at the right error output; how would you
approach debugging something like the following:
```shell
root@7fa702d1f4c4:/opt/airflow# pytest tests/charts/test_airflow_common.py
-n auto
============================================================================
test session starts
=============================================================================
platform linux -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0 --
/usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: timeouts-1.2.1, asyncio-0.21.0, capture-warnings-0.0.4,
httpx-0.21.3, time-machine-2.9.0, instafail-0.5.0, rerunfailures-11.1.2,
xdist-3.2.1, requests-mock-1.10.0, anyio-3.6.2, cov-4.0.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
[gw0] linux Python 3.7.16 cwd: /opt/airflow
[gw1] linux Python 3.7.16 cwd: /opt/airflow
[gw2] linux Python 3.7.16 cwd: /opt/airflow
[gw3] linux Python 3.7.16 cwd: /opt/airflow
[gw4] linux Python 3.7.16 cwd: /opt/airflow
[gw5] linux Python 3.7.16 cwd: /opt/airflow
[gw6] linux Python 3.7.16 cwd: /opt/airflow
[gw7] linux Python 3.7.16 cwd: /opt/airflow
[gw0] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw4] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw1] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw3] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw6] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw2] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw5] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
[gw7] Python 3.7.16 (default, Apr 12 2023, 07:12:00) -- [GCC 10.2.1
20210110]
gw0 [18] / gw1 [18] / gw2 [18] / gw3 [18] / gw4 [18] / gw5 [18] / gw6 [18] /
gw7 [18]
scheduling tests via LoadScheduling
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow@user-digest-user-tag-user-digest]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values0-expected_mount0]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_webserver_config_configmap_name_volume_mounts
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow:user-tag-user-tag-None]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow@user-digest-None-user-digest]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values2-expected_mount2]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_disable_some_variables
tests/charts/test_airflow_common.py::TestAirflowCommon::test_global_affinity_tolerations_topology_spread_constraints_and_node_selector
[gw0] [ 5%] FAILED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values0-expected_mount0]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values1-expected_mount1]
[gw4] [ 11%] FAILED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values2-expected_mount2]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values3-expected_mount3]
[gw1] [ 16%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_webserver_config_configmap_name_volume_mounts
tests/charts/test_airflow_common.py::TestAirflowCommon::test_annotations
[gw2] [ 22%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow:user-tag-user-tag-None]
[gw5] [ 27%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow@user-digest-user-tag-user-digest]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow@user-digest-None-user-digest]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_set_correct_helm_hooks_weight
[gw7] [ 33%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_disable_some_variables
[gw6] [ 38%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow@user-digest-None-user-digest]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_have_all_variables
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow@user-digest-user-tag-user-digest]
[gw0] [ 44%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values1-expected_mount1]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_have_all_config_mounts_on_init_containers
[gw4] [ 50%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values3-expected_mount3]
tests/charts/test_airflow_common.py::TestAirflowCommon::test_priority_class_name
[gw3] [ 55%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_global_affinity_tolerations_topology_spread_constraints_and_node_selector
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow:user-tag-user-tag-None]
[gw7] [ 61%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_have_all_variables
[gw6] [ 66%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow@user-digest-user-tag-user-digest]
[gw5] [ 72%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_set_correct_helm_hooks_weight
[gw2] [ 77%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_default_image[apache/airflow@user-digest-None-user-digest]
[gw0] [ 83%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_have_all_config_mounts_on_init_containers
[gw1] [ 88%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_annotations
[gw3] [ 94%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_should_use_correct_image[apache/airflow:user-tag-user-tag-None]
[gw4] [100%] PASSED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_priority_class_name
==================================================================================
FAILURES
==================================================================================
_______________________________________________________
TestAirflowCommon.test_dags_mount[dag_values0-expected_mount0]
_______________________________________________________
[gw0] linux -- Python 3.7.16 /usr/local/bin/python
self = <tests.charts.test_airflow_common.TestAirflowCommon object at
0x7ff0de8ec590>, dag_values = {'gitSync': {'enabled': True}}
expected_mount = {'mountPath': '/opt/airflow/dags', 'name': 'dags',
'readOnly': True}
@pytest.mark.parametrize(
"dag_values, expected_mount",
[
(
{"gitSync": {"enabled": True}},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": True,
},
),
(
{"persistence": {"enabled": True}},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": False,
},
),
(
{
"gitSync": {"enabled": True},
"persistence": {"enabled": True},
},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": True,
},
),
(
{"persistence": {"enabled": True, "subPath": "test/dags"}},
{
"subPath": "test/dags",
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": False,
},
),
],
)
def test_dags_mount(self, dag_values, expected_mount):
docs = render_chart(
values={
"dags": dag_values,
"airflowVersion": "1.10.15",
}, # airflowVersion is present so webserver gets the mount
show_only=[
"templates/scheduler/scheduler-deployment.yaml",
"templates/workers/worker-deployment.yaml",
> "templates/webserver/webserver-deployment.yaml",
],
)
tests/charts/test_airflow_common.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _
/home/mikaeld/code/mozilla/airflow/tests/charts/helm_template_generator.py:138:
in render_chart
???
/usr/local/lib/python3.7/subprocess.py:411: in check_output
**kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _
input = None, capture_output = False, timeout = None, check = True
popenargs = (['helm', 'template', 'release-name', '/opt/airflow/chart',
'--values', '/tmp/tmpjppcs087', ...],)
kwargs = {'cwd': '/opt/airflow/chart', 'stderr': -1, 'stdout': -1}, process
= <subprocess.Popen object at 0x7ff0dfc4ef90>, stdout = b''
stderr = b'W0417 15:54:23.300556 17785 loader.go:221] Config not found:
/files/.kube/config\nW0417 15:54:23.310400 17785 lo..." at
<.securityContexts.pod>: nil pointer evaluating interface {}.pod\n\nUse --debug
flag to render out invalid YAML\n'
retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False,
**kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout
and
stderr. By default, stdout and stderr are not captured, and those
attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to
capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the
return code
in the returncode attribute, and output & stderr attributes if those
streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this
argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input"
should
be bytes, and the stdout and stderr will be bytes. If in text mode,
any
"input" should be a string, and stdout and stderr will be strings
decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or
universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be
used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is
not None:
raise ValueError('stdout and stderr arguments may not be
used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
> output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['helm', 'template',
'release-name', '/opt/airflow/chart', '--values', '/tmp/tmpjppcs087',
'--kube-version', '1.23.13', '--namespace', 'default', '--show-only',
'templates/scheduler/scheduler-deployment.yaml', '--show-only',
'templates/workers/worker-deployment.yaml', '--show-only',
'templates/webserver/webserver-deployment.yaml']' returned non-zero exit status
1.
/usr/local/lib/python3.7/subprocess.py:512: CalledProcessError
_______________________________________________________
TestAirflowCommon.test_dags_mount[dag_values2-expected_mount2]
_______________________________________________________
[gw4] linux -- Python 3.7.16 /usr/local/bin/python
self = <tests.charts.test_airflow_common.TestAirflowCommon object at
0x7f8ec5f4ae90>, dag_values = {'gitSync': {'enabled': True}, 'persistence':
{'enabled': True}}
expected_mount = {'mountPath': '/opt/airflow/dags', 'name': 'dags',
'readOnly': True}
@pytest.mark.parametrize(
"dag_values, expected_mount",
[
(
{"gitSync": {"enabled": True}},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": True,
},
),
(
{"persistence": {"enabled": True}},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": False,
},
),
(
{
"gitSync": {"enabled": True},
"persistence": {"enabled": True},
},
{
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": True,
},
),
(
{"persistence": {"enabled": True, "subPath": "test/dags"}},
{
"subPath": "test/dags",
"mountPath": "/opt/airflow/dags",
"name": "dags",
"readOnly": False,
},
),
],
)
def test_dags_mount(self, dag_values, expected_mount):
docs = render_chart(
values={
"dags": dag_values,
"airflowVersion": "1.10.15",
}, # airflowVersion is present so webserver gets the mount
show_only=[
"templates/scheduler/scheduler-deployment.yaml",
"templates/workers/worker-deployment.yaml",
> "templates/webserver/webserver-deployment.yaml",
],
)
tests/charts/test_airflow_common.py:84:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _
/home/mikaeld/code/mozilla/airflow/tests/charts/helm_template_generator.py:138:
in render_chart
???
/usr/local/lib/python3.7/subprocess.py:411: in check_output
**kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _
input = None, capture_output = False, timeout = None, check = True
popenargs = (['helm', 'template', 'release-name', '/opt/airflow/chart',
'--values', '/tmp/tmpfd7jte59', ...],)
kwargs = {'cwd': '/opt/airflow/chart', 'stderr': -1, 'stdout': -1}, process
= <subprocess.Popen object at 0x7f8ec5be6910>, stdout = b''
stderr = b'W0417 15:54:23.307418 17790 loader.go:221] Config not found:
/files/.kube/config\nW0417 15:54:23.316569 17790 lo..." at
<.securityContexts.pod>: nil pointer evaluating interface {}.pod\n\nUse --debug
flag to render out invalid YAML\n'
retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False,
**kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout
and
stderr. By default, stdout and stderr are not captured, and those
attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to
capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the
return code
in the returncode attribute, and output & stderr attributes if those
streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this
argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input"
should
be bytes, and the stdout and stderr will be bytes. If in text mode,
any
"input" should be a string, and stdout and stderr will be strings
decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or
universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be
used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is
not None:
raise ValueError('stdout and stderr arguments may not be
used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
> output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['helm', 'template',
'release-name', '/opt/airflow/chart', '--values', '/tmp/tmpfd7jte59',
'--kube-version', '1.23.13', '--namespace', 'default', '--show-only',
'templates/scheduler/scheduler-deployment.yaml', '--show-only',
'templates/workers/worker-deployment.yaml', '--show-only',
'templates/webserver/webserver-deployment.yaml']' returned non-zero exit status
1.
/usr/local/lib/python3.7/subprocess.py:512: CalledProcessError
==========================================================================
short test summary info
===========================================================================
FAILED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values0-expected_mount0]
- subprocess.CalledProcessError: Command '['helm', 'template', 'release-name',
'/opt/airflow/chart', '--values', '/tmp/tmpjppcs087', '--kube-version',
'1.23.13', '--namespace', 'default', '--show-only',
'templates/scheduler/scheduler-deployment.yaml', '--show-only',
'templates/workers/worker-deployment.yaml', '--show-only',
'templates/webserver/webserver-deployment.yaml']' returned non-zero exit status
1.
FAILED
tests/charts/test_airflow_common.py::TestAirflowCommon::test_dags_mount[dag_values2-expected_mount2]
- subprocess.CalledProcessError: Command '['helm', 'template', 'release-name',
'/opt/airflow/chart', '--values', '/tmp/tmpfd7jte59', '--kube-version',
'1.23.13', '--namespace', 'default', '--show-only',
'templates/scheduler/scheduler-deployment.yaml', '--show-only',
'templates/workers/worker-deployment.yaml', '--show-only',
'templates/webserver/webserver-deployment.yaml']' returned non-zero exit status
1.
======================================================================= 2
failed, 16 passed in 30.49s
========================================================================
/usr/local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/__init__.py:105
MovedIn20Warning: Deprecated API features detected! These feature(s) are not
compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to
updating applications, ensure requirements files are pinned to
"sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all
deprecation warnings. Set environment variable
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on
SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
root@7fa702d1f4c4:/opt/airflow#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
