[
https://issues.apache.org/jira/browse/AIRFLOW-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952916#comment-16952916
]
ASF GitHub Bot commented on AIRFLOW-5126:
-----------------------------------------
ashb commented on pull request #6303: AIRFLOW-5126 Read aws_session_token in
extra_config of the aws hook
URL: https://github.com/apache/airflow/pull/6303
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Read aws_session_token in extra_config of the aws hook
> ------------------------------------------------------
>
> Key: AIRFLOW-5126
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5126
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
> Affects Versions: 1.10.3
> Reporter: Alexandre Blanchard
> Assignee: Johannes Günther
> Priority: Minor
>
> Hi,
> Thanks for the great software.
> At my company, we enforce security around our aws account and all accounts
> must have mfa activated. To use airflow with my account, I generate a session
> token with an expiration date using the command
> {code:java}
> aws sts assume-role --role-arn <the-role-i-want-use> --role-session-name
> testing --serial-number <my-personal-mfa-arn> --token-code
> <code-on-my-mfa-device>
> --duration-seconds 18000{code}
> This way I retrieve all I need to connect to aws: a aws_access_key_id, a
> aws_secret_access_key and a aws_session_token.
> Currently I'm using boto3 directly in my dag and it's working great. I would
> like to use a connection managed by airflow but when I set the parameters
> this way:
> {code:java}
> airflow connections --add \
> --conn_id s3_log \
> --conn_type s3 \
> --conn_login "<aws_access_key_id>" \
> --conn_password "<aws_secret_access_key>" \
> --conn_extra "{ \
> \"aws_session_token\": \"<aws_session_token>\" \
> }"
> {code}
> With a hook using this connection, I get the error:
> {code:java}
> [2019-08-06 12:31:28,157] {__init__.py:1580} ERROR - An error occurred (403)
> when calling the HeadObject operation: Forbidden
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/airflow/models/__init__.py",
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 112, in execute
> return_value = self.execute_callable()
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 117, in execute_callable
> return self.python_callable(*self.op_args, **self.op_kwargs)
> File "/root/airflow/dags/s3Dag.py", line 48, in download_raw_data
> dataObject = s3hook.get_key("poc/raw_data.csv.gz", s3_bucket)
> File "/usr/local/lib/python3.7/site-packages/airflow/hooks/S3_hook.py",
> line 217, in get_key
> obj.load()
> File "/usr/local/lib/python3.7/site-packages/boto3/resources/factory.py",
> line 505, in do_action
> response = action(self, *args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/boto3/resources/action.py",
> line 83, in __call__
> response = getattr(parent.meta.client, operation_name)(**params)
> File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357,
> in _api_call
> return self._make_api_call(operation_name, kwargs)
> File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661,
> in _make_api_call
> raise error_class(parsed_response, operation_name)
> botocore.exceptions.ClientError: An error occurred (403) when calling the
> HeadObject operation: Forbidden
> {code}
> Reading the code of the hook
> (https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/hooks/aws_hook.py#L90),
> I understand that the session token is not read from the extra config. The
> only case a session token is passed to the boto3 client is when we assume a
> role. In my case I want to use a role I have already assumed.
> So my suggestion is to read the session token from the extra config and use
> it to connect to aws.
> Do you think it is the right way to do it ? Does this workflow make sense ?
> I am ready to contribute if my suggestion is accepted.
> Regards
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
