xelita opened a new issue #20008: URL: https://github.com/apache/airflow/issues/20008
### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers <pre> Providers info apache-airflow-providers-amazon | 2.2.0 apache-airflow-providers-celery | 2.0.0 apache-airflow-providers-cncf-kubernetes | 2.0.2 apache-airflow-providers-docker | 2.1.1 apache-airflow-providers-elasticsearch | 2.0.3 apache-airflow-providers-ftp | 2.0.1 apache-airflow-providers-google | 5.1.0 apache-airflow-providers-grpc | 2.0.1 apache-airflow-providers-hashicorp | 2.1.0 apache-airflow-providers-http | 2.0.1 apache-airflow-providers-imap | 2.0.1 apache-airflow-providers-microsoft-azure | 3.1.1 apache-airflow-providers-mysql | 2.1.1 apache-airflow-providers-postgres | 2.2.0 apache-airflow-providers-redis | 2.0.1 apache-airflow-providers-sendgrid | 2.0.1 apache-airflow-providers-sftp | 2.1.1 apache-airflow-providers-slack | 4.0.1 apache-airflow-providers-sqlite | 2.0.1 apache-airflow-providers-ssh | 2.1.1 </pre> ### Apache Airflow version 2.1.4 ### Operating System Ubuntu 18.04.4 LTS ### Deployment Other Docker-based deployment ### Deployment details This is my current connection configuration to access AWS services using IAM role  ### What happened Recently, I have implemented a simple a DAG file that invokes a lambda function based on a schedule. To do that, I have defined an Airflow AWS connection just to set up the target AWS region - no other information is given there. I want to use EC2 instance metadata service to retrieve temporary aws credentials. The DAG execution worked fine until I started migrating to Instance Metadata Service Version 2 (IMDSv2) - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html And now I get the following errors: <pre> AIRFLOW_CTX_TASK_ID=invoke_lambda_task AIRFLOW_CTX_EXECUTION_DATE=2021-10-05T09:00:00+00:00 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-10-05T09:00:00+00:00 [2021-10-05 10:00:01,161] {base_aws.py:400} INFO - Airflow Connection: aws_conn_id=aws_default [2021-10-05 10:00:01,167] {base_aws.py:189} INFO - No credentials retrieved from Connection [2021-10-05 10:00:01,167] {base_aws.py:87} INFO - Retrieving region_name from Connection.extra_config['region_name'] [2021-10-05 10:00:01,167] {base_aws.py:92} INFO - Creating session with aws_access_key_id=None region_name=us-east-1 [2021-10-05 10:00:01,178] {base_aws.py:167} INFO - role_arn is None [2021-10-05 10:00:02,217] {taskinstance.py:1463} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.6/site-packages/airflow/operators/python.py", line 150, in execute return_value = self.execute_callable() File "/home/airflow/.local/lib/python3.6/site-packages/airflow/operators/python.py", line 161, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/opt/airflow/dags/notification-for-genosity-upload-to-repare.py", line 112, in invoke_lambda response = lambda_hook.invoke_lambda(payload="") File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/amazon/aws/hooks/lambda_function.py", line 66, in invoke_lambda Qualifier=self.qualifier, File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 386, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 692, in _make_api_call operation_model, request_dict, request_context) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/client.py", line 711, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request return self._send_request(request_dict, operation_model) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request request = self.create_request(request_dict, operation_model) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request operation_name=operation_model.name) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler return self.sign(operation_name, request) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/signers.py", line 162, in sign auth.add_auth(request) File "/home/airflow/.local/lib/python3.6/site-packages/botocore/auth.py", line 373, in add_auth raise NoCredentialsError() botocore.exceptions.NoCredentialsError: Unable to locate credentials [2021-10-05 10:00:02,219] {taskinstance.py:1513} INFO - Marking task as FAILED. dag_id=my_dag, task_id=invoke_lambda_task, execution_date=20211005T090000, start_date=20211005T100001, end_date=20211005T100002 [2021-10-05 10:00:02,274] {local_task_job.py:151} INFO - Task exited with return code 1 [2021-10-05 10:00:02,289] {local_task_job.py:261} INFO - 0 downstream tasks scheduled from follow-on schedule check </pre> It is like boto3 is not able to retrieve temporary credentials from IMDSv2... I am pretty sure something is missing in my configuration/DAG because boto3 documentation says it should seamlessly retrieve them from the instance metadata service... Meanwhile I have reverted my change to IMDSv1 which works fine but AWS highly recommends to migrate to v2... My DAG snippet that invokes the Lambda function Command that triggers AWS Lambda invocation: lambda_hook = AwsLambdaHook( aws_conn_id="aws_default", function_name="my-function", log_type="Tail" ) ### What you expected to happen Being able to call AWS services using IAM role and Instance Metadata Service Version 2 (IMDSv2) - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html ### How to reproduce _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
