Taragolis opened a new pull request, #38704:
URL: https://github.com/apache/airflow/pull/38704
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Add fallback to `region_name` it might not obtained if configuration not
persists in `airflow.cfg` or in env vars or configuration loaded without
provider configurations
In addition use `conf_vars` helper for configure tests, it will
automatically undo configuration in the end of test, so it reduce side effect
when one test allow to pass/fail next text tests.
Originally it cause flakey behaviour in parallel non-db tests when it might
run tests agains not properly configured tests in `main` or PRs
```console
________________ TestAwsBatchExecutor.test_health_check_failure
________________
[gw6] linux -- Python 3.12.2 /usr/local/bin/python
self =
<tests.providers.amazon.aws.executors.batch.test_batch_executor.TestAwsBatchExecutor
object at 0x7effd436bc20>
mock_executor = <MagicMock name='load' id='139637296591360'>
@mock.patch(
"airflow.providers.amazon.aws.executors.batch.boto_schema.BatchDescribeJobsResponseSchema.load"
)
def test_health_check_failure(self, mock_executor):
mock_executor.batch.describe_jobs.side_effect =
Exception("Test_failure")
> executor = AwsBatchExecutor()
tests/providers/amazon/aws/executors/batch/test_batch_executor.py:519:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
airflow/providers/amazon/aws/executors/batch/batch_executor.py:102: in
__init__
self.load_batch_connection(check_connection=False)
airflow/providers/amazon/aws/executors/batch/batch_executor.py:156: in
load_batch_connection
region_name = conf.get(CONFIG_GROUP_NAME, AllBatchConfigKeys.REGION_NAME)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _
self = <airflow.configuration.AirflowConfigParser object at 0x7f0002c10500>
section = 'aws_batch_executor', key = 'region_name', suppress_warnings =
False
_extra_stacklevel = 0, kwargs = {}, warning_emitted = False
option_description = {}, deprecated_section = None, deprecated_key = None
def get( # type: ignore[override,misc]
self,
section: str,
key: str,
suppress_warnings: bool = False,
_extra_stacklevel: int = 0,
**kwargs,
) -> str | None:
section = section.lower()
key = key.lower()
warning_emitted = False
deprecated_section: str | None
deprecated_key: str | None
option_description = self.configuration_description.get(section,
{}).get(key, {})
if option_description.get("deprecated"):
deprecation_reason =
option_description.get("deprecation_reason", "")
warnings.warn(
f"The '{key}' option in section {section} is deprecated.
{deprecation_reason}",
DeprecationWarning,
stacklevel=2 + _extra_stacklevel,
)
# For when we rename whole sections
if section in self.inversed_deprecated_sections:
deprecated_section, deprecated_key = (section, key)
section = self.inversed_deprecated_sections[section]
if not self._suppress_future_warnings:
warnings.warn(
f"The config section [{deprecated_section}] has been
renamed to "
f"[{section}]. Please update your `conf.get*` call to
use the new name",
FutureWarning,
stacklevel=2 + _extra_stacklevel,
)
# Don't warn about individual rename if the whole section is
renamed
warning_emitted = True
elif (section, key) in self.inversed_deprecated_options:
# Handle using deprecated section/key instead of the new
section/key
new_section, new_key =
self.inversed_deprecated_options[(section, key)]
if not self._suppress_future_warnings and not warning_emitted:
warnings.warn(
f"section/key [{section}/{key}] has been deprecated, you
should use"
f"[{new_section}/{new_key}] instead. Please update your
`conf.get*` call to use the "
"new name",
FutureWarning,
stacklevel=2 + _extra_stacklevel,
)
warning_emitted = True
deprecated_section, deprecated_key = section, key
section, key = (new_section, new_key)
elif section in self.deprecated_sections:
# When accessing the new section name, make sure we check under
the old config name
deprecated_key = key
deprecated_section = self.deprecated_sections[section][0]
else:
deprecated_section, deprecated_key, _ =
self.deprecated_options.get(
(section, key), (None, None, None)
)
# first check environment variables
option = self._get_environment_variables(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then the config file
option = self._get_option_from_config_file(
deprecated_key,
deprecated_section,
key,
kwargs,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then commands
option = self._get_option_from_commands(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then from secret backends
option = self._get_option_from_secrets(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then the default config
if self.get_default_value(section, key) is not None or "fallback" in
kwargs:
return expand_env_var(self.get_default_value(section, key,
**kwargs))
if self.get_default_pre_2_7_value(section, key) is not None:
# no expansion needed
return self.get_default_pre_2_7_value(section, key, **kwargs)
if not suppress_warnings:
log.warning("section/key [%s/%s] not found in config", section,
key)
> raise AirflowConfigException(f"section/key [{section}/{key}] not
found in config")
E airflow.exceptions.AirflowConfigException: section/key
[aws_batch_executor/region_name] not found in config
airflow/configuration.py:1052: AirflowConfigException
----------------------------- Captured stdout call
-----------------------------
[2024-04-03T10:14:13.235+0000] {batch_executor.py:150} INFO - Loading
Connection information
[2024-04-03T10:14:13.237+0000] {configuration.py:1050} WARNING - section/***
[aws_batch_executor/region_name] not found in config
FAILED
tests/providers/amazon/aws/executors/batch/test_batch_executor.py::TestAwsBatchExecutor::test_health_check_failure
- airflow.exceptions.AirflowConfigException: section/key
[aws_batch_executor/region_name] not found in config
```
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
