[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-08-06 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r466649936



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, 
LoggingMixin):
 
 def __init__(
 self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,

Review comment:
   The following should work in your case (can you try):
   
   ```ini
   [secrets]
   backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
   backend_kwargs = {"connections_prefix": "", "variables_prefix": "", "sep": 
""}
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-08-06 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r466649936



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, 
LoggingMixin):
 
 def __init__(
 self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,

Review comment:
   The following should work in your case (can you try):
   
   ```ini
   [secrets]
   backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend
   backend_kwargs = {"connections_prefix": "", "sep": ""}
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455698455



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, 
LoggingMixin):
 
 def __init__(
 self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,
+variables_prefix=None,

Review comment:
   it does. If someone passes nothing, the default it used to take is 
`airflow/variables`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455698455



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, 
LoggingMixin):
 
 def __init__(
 self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,
+variables_prefix=None,

Review comment:
   it does. If no-one passes anything, the default it used to take is 
`airflow/variables`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455691530



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -83,43 +90,121 @@ def client(self):
 )
 return session.client(service_name="secretsmanager", **self.kwargs)
 
-def get_conn_uri(self, conn_id: str) -> Optional[str]:
+def _get_user_and_password(self, secret):
+for user_denomination in ['user', 'username', 'login']:
+if user_denomination in secret:
+user = secret[user_denomination]
+
+for password_denomination in ['pass', 'password', 'key']:
+if password_denomination in secret:
+password = secret[password_denomination]
+
+return user, password
+
+def _get_host(self, secret):
+for host_denomination in ['host', 'remote_host', 'server']:
+if host_denomination in secret:
+host = secret[host_denomination]
+
+return host
+
+def _get_conn_type(self, secret):
+for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']:
+if type_word in secret:
+conn_type = secret[type_word]
+conn_type = 'postgresql' if conn_type == 'redshift' else 
conn_type
+else:
+conn_type = 'connection'
+return conn_type
+
+def _get_port_and_database(self, secret):
+if 'port' in secret:
+port = secret['port']
+else:
+port = 5432

Review comment:
   Check 
https://github.com/apache/airflow/blob/master/airflow/secrets/local_filesystem.py
 out





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455691248



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -83,43 +90,121 @@ def client(self):
 )
 return session.client(service_name="secretsmanager", **self.kwargs)
 
-def get_conn_uri(self, conn_id: str) -> Optional[str]:
+def _get_user_and_password(self, secret):
+for user_denomination in ['user', 'username', 'login']:
+if user_denomination in secret:
+user = secret[user_denomination]
+
+for password_denomination in ['pass', 'password', 'key']:
+if password_denomination in secret:
+password = secret[password_denomination]
+
+return user, password
+
+def _get_host(self, secret):
+for host_denomination in ['host', 'remote_host', 'server']:
+if host_denomination in secret:
+host = secret[host_denomination]
+
+return host
+
+def _get_conn_type(self, secret):
+for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']:
+if type_word in secret:
+conn_type = secret[type_word]
+conn_type = 'postgresql' if conn_type == 'redshift' else 
conn_type
+else:
+conn_type = 'connection'
+return conn_type
+
+def _get_port_and_database(self, secret):
+if 'port' in secret:
+port = secret['port']
+else:
+port = 5432

Review comment:
   Let's not hardcode or keep any default here





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455691248



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -83,43 +90,121 @@ def client(self):
 )
 return session.client(service_name="secretsmanager", **self.kwargs)
 
-def get_conn_uri(self, conn_id: str) -> Optional[str]:
+def _get_user_and_password(self, secret):
+for user_denomination in ['user', 'username', 'login']:
+if user_denomination in secret:
+user = secret[user_denomination]
+
+for password_denomination in ['pass', 'password', 'key']:
+if password_denomination in secret:
+password = secret[password_denomination]
+
+return user, password
+
+def _get_host(self, secret):
+for host_denomination in ['host', 'remote_host', 'server']:
+if host_denomination in secret:
+host = secret[host_denomination]
+
+return host
+
+def _get_conn_type(self, secret):
+for type_word in ['conn_type', 'conn_id', 'connection_type', 'engine']:
+if type_word in secret:
+conn_type = secret[type_word]
+conn_type = 'postgresql' if conn_type == 'redshift' else 
conn_type
+else:
+conn_type = 'connection'
+return conn_type
+
+def _get_port_and_database(self, secret):
+if 'port' in secret:
+port = secret['port']
+else:
+port = 5432

Review comment:
   Let's not hardcode 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-07-16 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r455690869



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -60,15 +59,23 @@ class SecretsManagerBackend(BaseSecretsBackend, 
LoggingMixin):
 
 def __init__(
 self,
-connections_prefix: str = 'airflow/connections',
-variables_prefix: str = 'airflow/variables',
-profile_name: Optional[str] = None,
-sep: str = "/",
+connections_prefix=None,
+variables_prefix=None,

Review comment:
   It also breaks backwards-compatibility





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [airflow] kaxil commented on a change in pull request #9008: Get connections uri with AWS Secrets Manager backend

2020-05-29 Thread GitBox


kaxil commented on a change in pull request #9008:
URL: https://github.com/apache/airflow/pull/9008#discussion_r432544185



##
File path: airflow/providers/amazon/aws/secrets/secrets_manager.py
##
@@ -83,43 +86,87 @@ def client(self):
 )
 return session.client(service_name="secretsmanager", **self.kwargs)
 
-def get_conn_uri(self, conn_id: str) -> Optional[str]:
+def get_conn_uri(self, conn_id: str):
 """
 Get Connection Value
 
 :param conn_id: connection id
 :type conn_id: str
 """
-return self._get_secret(self.connections_prefix, conn_id)
+if self.connections_prefix and self.sep:
+conn_id = self.build_path(self.connections_prefix, conn_id, 
self.sep)
 
-def get_variable(self, key: str) -> Optional[str]:
+try:
+secret_string = self._get_secret(conn_id)
+secret = ast.literal_eval(secret_string)
+except ValueError:  # 'malformed node or string: ' error, for empty 
conns
+connection = None
+
+# These lines will check if we have with some denomination stored an 
username, password and host
+if secret:
+for user_denomination in ['user', 'username', 'login']:

Review comment:
   yes avoid the nested blocks if possible, if it is unavoidable then just 
add `# pylint: disable=too-many-nested-blocks` at the end of the line where it 
complains.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org