Arthur Wiedmer created AIRFLOW-115:
--------------------------------------
Summary: Migrate and Refactor AWS integration to use boto3 and
better structured hooks
Key: AIRFLOW-115
URL: https://issues.apache.org/jira/browse/AIRFLOW-115
Project: Apache Airflow
Issue Type: Improvement
Components: AWS, boto3, hooks
Reporter: Arthur Wiedmer
Priority: Minor
h2. Current State
The current AWS integration is mostly done through the S3Hook, which uses non
standard credentials parsing on top of using boto instead of boto3 which is the
current supported AWS sdk for Python.
h2. Proposal
an AWSHook should be provided that maps Airflow connections to the boto3 API.
Operators working with s3, as well as other AWS services would then inherit
from this hook but extend the functionality with service specific methods like
get_key for S3, start_cluster for EMR, enqueue for SQS, send_email for SES
etc...
AWSHook
|_S3Hook
|_EMRHook
|_SQSHook
|_SESHook
...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
