Taragolis commented on issue #27078: URL: https://github.com/apache/airflow/issues/27078#issuecomment-1282595490
@btoh You can't use botocore.Config for setup `endpoint_url`, see https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html `endpoint_url` (old version of provider use `host` instead) uses only in client which created by hook but not during obtain credential by STS AssumeRole. Simplify version what happen internally when you try to assume role ```python import boto3 botocore_config = botocore.config.Config(...) role_arn = "arn:aws:iam::123456789012:role/demo" endpoint_url = "elasticmapreduce.eu-west-2.amazonaws.com" region_name = "eu-west-2" # Create initial boto3 session initial_session = boto3.session.Session(region_name=region_name) # Create sts client from initial session sts_client = initial_session.client("sts", config=botocore_config) # Assume role and get credentials, in airflow this process a bit different creds = sts_client.assume_role( RoleArn=role_arn, ... )["Credentials"] # Target boto3 session, in airflow this process a bit different session = boto3.session.Session( aws_access_key_id=credentials["AccessKeyId"], aws_secret_access_key=credentials["SecretAccessKey"], aws_session_token=credentials["SessionToken"], region_name=region_name, ) # Create target client, e.g. EMR target_client = session.client("emr", config=botocore_config, endpoint_url=endpoint_url) ``` STS service use global endpoint `https://sts.amazonaws.com` which not depend on actual `region_name`, most of other clients use in general regional endpoints for AWS API calls which depend on `region_name`, e.g. `https://s3.eu-west-2.amazonaws.com` Right not it not make sense to pass to sts client anything which not support STS API calls, see some sample ```python Python 3.9.9 (main, Jan 5 2022, 16:02:57) [Clang 13.0.0 (clang-1300.0.27.3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import boto3 >>> session = boto3.session.Session(region_name="eu-west-2") >>> session Session(region_name='eu-west-2') >>> sts_client = session.client("sts") >>> sts_client <botocore.client.STS object at 0x102e8f730> >>> sts_client.meta.endpoint_url 'https://sts.amazonaws.com' >>> emr_client = session.client("emr") ../.nox/develop/lib/python3.9/site-packages/botocore/client.py:625: FutureWarning: The elasticmapreduce client is currently using a deprecated endpoint: eu-west-2.elasticmapreduce.amazonaws.com. In the next minor version this will be moved to elasticmapreduce.eu-west-2.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details. >>> emr_client.meta.endpoint_url 'https://eu-west-2.elasticmapreduce.amazonaws.com' >>> session.client("s3").meta.endpoint_url 'https://s3.eu-west-2.amazonaws.com' >>> sts_client_2 = session.client("sts", endpoint_url="https://elasticmapreduce.eu-west-2.amazonaws.com";) >>> sts_client_2.meta.endpoint_url 'https://elasticmapreduce.eu-west-2.amazonaws.com' >>> # Lets call some STS Api calls (for both clients) >>> sts_client.get_caller_identity() {'UserId': 'REDACTED', 'Account': 'REDACTED', 'Arn': 'REDACTED', 'ResponseMetadata': {...}, 'RetryAttempts': 0}} >>> sts_client_2.get_caller_identity() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "../.nox/develop/lib/python3.9/site-packages/botocore/client.py", line 514, in _api_call return self._make_api_call(operation_name, kwargs) File "../.nox/develop/lib/python3.9/site-packages/botocore/client.py", line 938, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InvalidAction) when calling the GetCallerIdentity operation: Could not find operation GetCallerIdentity for version 2011-06-15 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
