Adrian Partl created AIRFLOW-1023:
-------------------------------------
Summary: Upload file to S3 using S3 hook fails with "Connection
reset by peer"
Key: AIRFLOW-1023
URL: https://issues.apache.org/jira/browse/AIRFLOW-1023
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: Airflow 1.7.1
Reporter: Adrian Partl
Assignee: Adrian Partl
Using the S3 hook to upload local files to an S3 bucket fails with
{noformat}
File "/usr/lib/python2.7/site-packages/airflow/hooks/S3_hook.py", line 364,
in load_file
replace=replace)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1362, in
set_contents_from_filename
encrypt_key=encrypt_key)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in
set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in
_send_file_internal
query_args=query_args
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in
make_request
retry_handler=retry_handler
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in
make_request
retry_handler=retry_handler)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in
_mexe
raise ex
error: [Errno 104] Connection reset by peer
{noformat}
This is a known issue with boto and only affects uploads to S3 buckets outside
of the standard US location (in my case {{eu-west-1}}).
The issue is reported on boto side as:
https://github.com/boto/boto/issues/2207
A work around is mentioned by user {{anna-buttfield-sirca}} which basically
reconnects the boto S3 connection to the corresponding location.
I will provide a PR implementing the work around, since a resolution of the
issue on the boto side seems unlikely.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
