Flogue opened a new issue, #26754:
URL: https://github.com/apache/airflow/issues/26754
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
Providers included in apache/airflow:2.4.0 docker image:
```
apache-airflow==2.4.0
apache-airflow-providers-amazon==5.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-mysql==3.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0
```
### Apache Airflow version
2.4
### Operating System
AWS Fargate
### Deployment
Docker-Compose
### Deployment details
We have configure the alternative secrets backend to use AWS SMP:
```
[secrets]
backend =
airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"config_prefix": "/airflow2/config", "connections_prefix":
"/airflow2/connections", "variables_prefix": "/airflow2/variables"}
```
### What happened
```
All processes fail with:
`Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 178, in _get_secret
response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
File
"/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36,
in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from
'airflow.providers.amazon.aws.hooks.base_aws'
(/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py",
line 122, in _get_config_value_from_secret_backend
return secrets_client.get_config(config_key)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 167, in get_config
return self._get_secret(self.config_prefix, key)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 180, in _get_secret
except self.client.exceptions.ParameterNotFound:
File
"/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36,
in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from
'airflow.providers.amazon.aws.hooks.base_aws'
(/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 178, in _get_secret
response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
File
"/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36,
in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py",
line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py",
line 49, in <module>
from airflow.models.connection import Connection
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py",
line 32, in <module>
from airflow.models.base import ID_LEN, Base
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line
76, in <module>
COLLATION_ARGS = get_id_collation_args()
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line
70, in get_id_collation_args
conn = conf.get('database', 'sql_alchemy_conn', fallback='')
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py",
line 557, in get
option = self._get_option_from_secrets(deprecated_key,
deprecated_section, key, section)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py",
line 577, in _get_option_from_secrets
option = self._get_secret_option(section, key)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py",
line 502, in _get_secret_option
return _get_config_value_from_secret_backend(secrets_path)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py",
line 125, in _get_config_value_from_secret_backend
'Cannot retrieve config from alternative secrets backend. '
airflow.exceptions.AirflowConfigException: Cannot retrieve config from
alternative secrets backend. Make sure it is configured properly and that the
Backend is accessible.
cannot import name 'SessionFactory' from
'airflow.providers.amazon.aws.hooks.base_aws'
(/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
```
`
### What you think should happen instead
We are trying to import 'SessionFactory' from
'airflow.providers.amazon.aws.hooks.base_aws' but only BaseSessionFactory is
defined in there.
### How to reproduce
I assume anyone that sets up the parameter store as backend and try to use
the docker image (FROM apache/airflow:2.4.0) will run into this issue.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
