After some more research it appears that aws_hook in contrib/hooks actually
does do it the way I proposed with looking at the login and password of the
connection, but it doesn’t look at the extra json for the access and secret
keys.
def get_client_type(self, client_type, region_name=None):
try:
connection_object = self.get_connection(self.aws_conn_id)
aws_access_key_id = connection_object.login
aws_secret_access_key = connection_object.password
if region_name is None:
region_name = connection_object.extra_dejson.get('region_name')
except AirflowException:
# No connection found: fallback on boto3 credential strategy
# http://boto3.readthedocs.io/en/latest/guide/configuration.html
aws_access_key_id = None
aws_secret_access_key = None
return boto3.client(
client_type,
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key
)
However the S3Hook looks for this info in a different using the older boto
library instead of boto3.
So it appears we have 2 different parts of airflow interacting with aws but
specifying their credentials in different ways.
Thoughts?
> On Sep 19, 2017, at 12:01 PM, Ali Uz <[email protected]> wrote:
>
> We use a dynamic config where we iterate through a JSON file, and all
> sensitive info (like api keys, aws keys, etc...) are pulled from a remote
> k/v store when airflow starts and adds them as fields to the JSON config
> file.
>
> On Tue, Sep 19, 2017 at 6:54 PM, Michael Crawford <
> [email protected]> wrote:
>
>> Did my message go through? I have never tried to send an email to the
>> list before, only silently monitored.
>>
>> Does anyone have any ideas? I would be happy to create an issue and code
>> up the fix myself, but I just wanted to ping here first to make sure I
>> wasn’t missing anything and try to get a consensus on how to handle this.
>>
>> Thanks,
>> Mike
>>
>>> On Sep 18, 2017, at 8:03 PM, Michael Crawford <michael.crawford@
>> modernizingmedicine.com> wrote:
>>>
>>> Hi,
>>>
>>> I was wondering if anything had ever been proposed for having the aws
>> secret key hidden in the aws type connection.
>>>
>>> Currently passing in these credentials is done by defining the some json
>> in the extra params section of the connection like
>>> {"aws_access_key_id":"_your_aws_access_key_id_",
>> "aws_secret_access_key": "_your_aws_secret_access_key_”}
>>>
>>> While this does work it leaves the secret access key in plain text for
>> anyone that has access to the connections.
>>>
>>> I know there are other options about setting them as environment
>> variables, but this doesn’t help if we need to define more than one aws
>> connection with different access keys.
>>>
>>> Two things that immediately came to mind for how to do this:
>>>
>>> 1. use login and password sections of the connection for the access and
>> secret keys so that the secret gets hidden and encrypted like all the other
>> passwords.
>>> 2. have an option to encrypt the extra params
>>>
>>> Option 1 seems most logical and should be too hard to implement.
>>>
>>> Open to any ideas people might have on this.
>>>
>>> Thanks,
>>> Mike
>>
>>
