[
https://issues.apache.org/jira/browse/AIRFLOW-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack updated AIRFLOW-2999:
--------------------------
Description:
The [S3_hook
|https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177]
has get_key method that returns boto3.s3.Object it also has load_file method
which loads file from local file system to S3.
What it doesn't have is a method to download a file from S3 to the local file
system.
Basicly it should be something very simple... an extention to the get_key
method with parameter to the destination on local file system adding a code for
taking the boto3.s3.Object and save it on the disk. Note: that it can be more
than 1 file if the user choose a folder in S3.
+*Update:*+
As discussed in comments instead having the property in the hook it's better to
mirror the GoogleCloudStorageDownloadOperator and have S3DownloadOperator
was:
The [S3_hook
|https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177]
has get_key method that returns boto3.s3.Object it also has load_file method
which loads file from local file system to S3.
What it doesn't have is a method to download a file from S3 to the local file
system.
Basicly it should be something very simple... an extention to the get_key
method with parameter to the destination on local file system adding a code for
taking the boto3.s3.Object and save it on the disk. Note: that it can be more
than 1 file if the user choose a folder in S3.
> Add S3DownloadOperator
> ----------------------
>
> Key: AIRFLOW-2999
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2999
> Project: Apache Airflow
> Issue Type: Task
> Affects Versions: 1.10.0
> Reporter: jack
> Assignee: Tanay Tummalapalli
> Priority: Major
>
> The [S3_hook
> |https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/S3_hook.py#L177]
> has get_key method that returns boto3.s3.Object it also has load_file method
> which loads file from local file system to S3.
>
> What it doesn't have is a method to download a file from S3 to the local file
> system.
> Basicly it should be something very simple... an extention to the get_key
> method with parameter to the destination on local file system adding a code
> for taking the boto3.s3.Object and save it on the disk. Note: that it can be
> more than 1 file if the user choose a folder in S3.
>
> +*Update:*+
> As discussed in comments instead having the property in the hook it's better
> to mirror the GoogleCloudStorageDownloadOperator and have S3DownloadOperator
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
