dstandish opened a new pull request #19504:
URL: https://github.com/apache/airflow/pull/19504
When you download a key that exists, notice that it retrieves creds twice:
```
[2021-11-09 22:25:18,736] {s3.py:807} INFO - Downloading source S3 file from
Bucket oss-test-xcom with path
test-dag/test-task/2021-01-01T00:00:00+00:00/hellodf
[2021-11-09 22:25:18,736] {base_aws.py:401} INFO - Airflow Connection:
aws_conn_id=aws_default
[2021-11-09 22:25:18,752] {credentials.py:1224} INFO - Found credentials in
shared credentials file: ~/.aws/credentials
[2021-11-09 22:25:19,049] {base_aws.py:424} WARNING - Unable to use Airflow
Connection for credentials.
[2021-11-09 22:25:19,049] {base_aws.py:425} INFO - Fallback on boto3
credential strategy
[2021-11-09 22:25:19,049] {base_aws.py:428} INFO - Creating session using
boto3 credential strategy region_name=None
/Users/dstandish/code/airflow/airflow/providers/amazon/aws/hooks/base_aws.py:494
DeprecationWarning: client_type is deprecated. Set client_type from class
attribute.
[2021-11-09 22:25:19,066] {credentials.py:1224} INFO - Found credentials in
shared credentials file: ~/.aws/credentials
[2021-11-09 22:25:19,526] {base_aws.py:401} INFO - Airflow Connection:
aws_conn_id=aws_default
[2021-11-09 22:25:19,594] {base_aws.py:424} WARNING - Unable to use Airflow
Connection for credentials.
[2021-11-09 22:25:19,594] {base_aws.py:425} INFO - Fallback on boto3
credential strategy
[2021-11-09 22:25:19,594] {base_aws.py:428} INFO - Creating session using
boto3 credential strategy region_name=None
/Users/dstandish/code/airflow/airflow/providers/amazon/aws/hooks/s3.py:343
DeprecationWarning: resource_type is deprecated. Set resource_type from class
attribute.
[2021-11-09 22:25:19,628] {credentials.py:1224} INFO - Found credentials in
shared credentials file: ~/.aws/credentials
```
The first is for checking existence and the second is retrieving the object.
We don't need to check for existence. We can just ask for the object and if
it's not there, the api will let is know. And when the object _is_ there,
we'll only have retrieved creds once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
