[
https://issues.apache.org/jira/browse/AIRFLOW-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928529#comment-15928529
]
Adamos Loizou commented on AIRFLOW-115:
---------------------------------------
+1 on this one.
Airflow's S3Hook is also problematic when using non-US S3 buckets such as
eu-central-1 which also require v4 S3 communication API.
Because of this we had to scrap using Airflow plugins and code it directly
ourselves as a python app.
It would be great if Airflow could upgrade boto and allow for more
configuration to be passed down to the plugin.
> Migrate and Refactor AWS integration to use boto3 and better structured hooks
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-115
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, boto3, hooks
> Reporter: Arthur Wiedmer
> Assignee: Arthur Wiedmer
> Priority: Minor
>
> h2. Current State
> The current AWS integration is mostly done through the S3Hook, which uses non
> standard credentials parsing on top of using boto instead of boto3 which is
> the current supported AWS sdk for Python.
> h2. Proposal
> an AWSHook should be provided that maps Airflow connections to the boto3 API.
> Operators working with s3, as well as other AWS services would then inherit
> from this hook but extend the functionality with service specific methods
> like get_key for S3, start_cluster for EMR, enqueue for SQS, send_email for
> SES etc...
> * AWSHook
> ** S3Hook
> ** EMRHook
> ** SQSHook
> ** SESHook
> ...
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
