[
https://issues.apache.org/jira/browse/AIRFLOW-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601582#comment-16601582
]
Apache Spark commented on AIRFLOW-1023:
---------------------------------------
User 'adrpar' has created a pull request for this issue:
https://github.com/apache/incubator-airflow/pull/2176
> Upload file to S3 using S3 hook fails with "Connection reset by peer"
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-1023
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1023
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: Airflow 1.7.1
> Reporter: Adrian Partl
> Assignee: Adrian Partl
> Priority: Major
>
> Using the S3 hook to upload local files to an S3 bucket fails with
> {noformat}
> File "/usr/lib/python2.7/site-packages/airflow/hooks/S3_hook.py", line 364,
> in load_file
> replace=replace)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1362, in
> set_contents_from_filename
> encrypt_key=encrypt_key)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in
> set_contents_from_file
> chunked_transfer=chunked_transfer, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in
> send_file
> chunked_transfer=chunked_transfer, size=size)
> File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in
> _send_file_internal
> query_args=query_args
> File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in
> make_request
> retry_handler=retry_handler
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in
> make_request
> retry_handler=retry_handler)
> File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in
> _mexe
> raise ex
> error: [Errno 104] Connection reset by peer
> {noformat}
> This is a known issue with boto and only affects uploads to S3 buckets
> outside of the standard US location (in my case {{eu-west-1}}).
> The issue is reported on boto side as:
> https://github.com/boto/boto/issues/2207
> A work around is mentioned by user {{anna-buttfield-sirca}} which basically
> reconnects the boto S3 connection to the corresponding location.
> I will provide a PR implementing the work around, since a resolution of the
> issue on the boto side seems unlikely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
