[
https://issues.apache.org/jira/browse/AIRFLOW-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197263#comment-16197263
]
Josh Bacon commented on AIRFLOW-115:
------------------------------------
Hey all!
Another reason to upgrade to boto3, yet to be mentioned here, is enabled
support for IAM roles. My team was evaluating Airflow's [remote s3 logging
feature|https://airflow.incubator.apache.org/configuration.html#logs] in order
to avoid exposing a log server from our Celery Workers. Unfortunately we
abandoned the built-in log functionality since there is no support for IAM with
the s3_hook and we are not permitted to use permanent AWS access keys.
Maybe something more to think about, thanks for all the great contributions
here!
> Migrate and Refactor AWS integration to use boto3 and better structured hooks
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-115
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, boto3, hooks
> Reporter: Arthur Wiedmer
> Assignee: Arthur Wiedmer
> Priority: Minor
>
> h2. Current State
> The current AWS integration is mostly done through the S3Hook, which uses non
> standard credentials parsing on top of using boto instead of boto3 which is
> the current supported AWS sdk for Python.
> h2. Proposal
> an AWSHook should be provided that maps Airflow connections to the boto3 API.
> Operators working with s3, as well as other AWS services would then inherit
> from this hook but extend the functionality with service specific methods
> like get_key for S3, start_cluster for EMR, enqueue for SQS, send_email for
> SES etc...
> * AWSHook
> ** S3Hook
> ** EMRHook
> ** SQSHook
> ** SESHook
> ...
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
