[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r466649936
##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend,
LoggingMixin):
def __init__(
self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,
Review comment:
The following should work in your case (can you try):
```ini
[secrets]
backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "", "variables_prefix": "", "sep":
""}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r466649936
##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend,
LoggingMixin):
def __init__(
self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,
Review comment:
The following should work in your case (can you try):
```ini
[secrets]
backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "", "sep": ""}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455698455 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin): def __init__( self, -connections_prefix: str = 'airflow/connections', -variables_prefix: str = 'airflow/variables', -profile_name: Optional[str] = None, -sep: str = "/", +connections_prefix=None, +variables_prefix=None, Review comment: it does. If someone passes nothing, the default it used to take is `airflow/variables` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455698455 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin): def __init__( self, -connections_prefix: str = 'airflow/connections', -variables_prefix: str = 'airflow/variables', -profile_name: Optional[str] = None, -sep: str = "/", +connections_prefix=None, +variables_prefix=None, Review comment: it does. If no-one passes anything, the default it used to take is `airflow/variables` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455691530 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -83,43 +90,121 @@ def client(self): ) return session.client(service_name="secretsmanager", **self.kwargs) -def get_conn_uri(self, conn_id: str) -> Optional[str]: +def _get_user_and_password(self, secret): +for user_denomination in ['user', 'username', 'login']: +if user_denomination in secret: +user = secret[user_denomination] + +for password_denomination in ['pass', 'password', 'key']: +if password_denomination in secret: +password = secret[password_denomination] + +return user, password + +def _get_host(self, secret): +for host_denomination in ['host', 'remote_host', 'server']: +if host_denomination in secret: +host = secret[host_denomination] + +return host + +def _get_conn_type(self, secret): +for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']: +if type_word in secret: +conn_type = secret[type_word] +conn_type = 'postgresql' if conn_type == 'redshift' else conn_type +else: +conn_type = 'connection' +return conn_type + +def _get_port_and_database(self, secret): +if 'port' in secret: +port = secret['port'] +else: +port = 5432 Review comment: Check https://github.com/apache/airflow/blob/master/airflow/secrets/local_filesystem.py out This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455691248 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -83,43 +90,121 @@ def client(self): ) return session.client(service_name="secretsmanager", **self.kwargs) -def get_conn_uri(self, conn_id: str) -> Optional[str]: +def _get_user_and_password(self, secret): +for user_denomination in ['user', 'username', 'login']: +if user_denomination in secret: +user = secret[user_denomination] + +for password_denomination in ['pass', 'password', 'key']: +if password_denomination in secret: +password = secret[password_denomination] + +return user, password + +def _get_host(self, secret): +for host_denomination in ['host', 'remote_host', 'server']: +if host_denomination in secret: +host = secret[host_denomination] + +return host + +def _get_conn_type(self, secret): +for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']: +if type_word in secret: +conn_type = secret[type_word] +conn_type = 'postgresql' if conn_type == 'redshift' else conn_type +else: +conn_type = 'connection' +return conn_type + +def _get_port_and_database(self, secret): +if 'port' in secret: +port = secret['port'] +else: +port = 5432 Review comment: Let's not hardcode or keep any default here This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455691248 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -83,43 +90,121 @@ def client(self): ) return session.client(service_name="secretsmanager", **self.kwargs) -def get_conn_uri(self, conn_id: str) -> Optional[str]: +def _get_user_and_password(self, secret): +for user_denomination in ['user', 'username', 'login']: +if user_denomination in secret: +user = secret[user_denomination] + +for password_denomination in ['pass', 'password', 'key']: +if password_denomination in secret: +password = secret[password_denomination] + +return user, password + +def _get_host(self, secret): +for host_denomination in ['host', 'remote_host', 'server']: +if host_denomination in secret: +host = secret[host_denomination] + +return host + +def _get_conn_type(self, secret): +for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']: +if type_word in secret: +conn_type = secret[type_word] +conn_type = 'postgresql' if conn_type == 'redshift' else conn_type +else: +conn_type = 'connection' +return conn_type + +def _get_port_and_database(self, secret): +if 'port' in secret: +port = secret['port'] +else: +port = 5432 Review comment: Let's not hardcode This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r455690869 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin): def __init__( self, -connections_prefix: str = 'airflow/connections', -variables_prefix: str = 'airflow/variables', -profile_name: Optional[str] = None, -sep: str = "/", +connections_prefix=None, +variables_prefix=None, Review comment: It also breaks backwards-compatibility This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend
kaxil commented on a change in pull request #9008: URL: https://github.com/apache/airflow/pull/9008#discussion_r432544185 ## File path: airflow/providers/amazon/aws/secrets/secrets_manager.py ## @@ -83,43 +86,87 @@ def client(self): ) return session.client(service_name="secretsmanager", **self.kwargs) -def get_conn_uri(self, conn_id: str) -> Optional[str]: +def get_conn_uri(self, conn_id: str): """ Get Connection Value :param conn_id: connection id :type conn_id: str """ -return self._get_secret(self.connections_prefix, conn_id) +if self.connections_prefix and self.sep: +conn_id = self.build_path(self.connections_prefix, conn_id, self.sep) -def get_variable(self, key: str) -> Optional[str]: +try: +secret_string = self._get_secret(conn_id) +secret = ast.literal_eval(secret_string) +except ValueError: # 'malformed node or string: ' error, for empty conns +connection = None + +# These lines will check if we have with some denomination stored an username, password and host +if secret: +for user_denomination in ['user', 'username', 'login']: Review comment: yes avoid the nested blocks if possible, if it is unavoidable then just add `# pylint: disable=too-many-nested-blocks` at the end of the line where it complains. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
