GitHub user amisstea opened a pull request:
https://github.com/apache/libcloud/pull/1005
[LIBCLOUD-834] Reintroduce S3 multipart upload support with signature v4
## Reintroduce S3 multipart upload support with signature v4
### Description
All S3 regions now support signature V4. Rather than try to implement
multipart
uploads with both V2 and V4, all region drivers have been converted to use
the latter. All multipart changes are based on code which existed in v1.5.0.
Some minor adjustments were made to the use of requests made throughout
the BaseS3StorageDriver class. The practice of constructing request paths
with
parameters included needed to be removed. It causes issues during the
signature calculation. The solution is to use the params argument instead.
In addition, a fix was necessary in the signature calculation to not
compute a
payload hash in the event of file uploads. It seems this was already taken
into
consideration but failed to account for the scenario where an iterator is
passed to the data argument.
I have tested these changes against S3 using each of the following methods
and with each regional driver.
get_container
create_container
delete_container
list_container_objects
get_object
delete_object
upload_object
upload_object_via_stream
download_object
download_object_via_stream
ex_iterate_multipart_uploads
ex_cleanup_all_multipart_uploads
### Status
- done, ready for review
### Checklist (tick everything that applies)
- [x] [Code
linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide)
(required, can be done after the PR checks)
- [x] Documentation
- [x] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
- [ ]
[ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes)
(required for bigger changes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amisstea/libcloud
libcloud-834-s3-multipart-upload-v4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/libcloud/pull/1005.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1005
----
commit 01967258b3ae5f4307ca352180d1098c10209bbb
Author: Alex Misstear <[email protected]>
Date: 2017-03-08T21:29:27Z
[LIBCLOUD-834] Reintroduce S3 multipart upload support with signature v4
All S3 regions now support signature V4. Rather than try to implement
multipart
uploads with both V2 and V4, all region drivers have been converted to use
the latter. All multipart changes are based on code which existed in v1.5.0.
Some minor adjustments were made to the use of requests made throughout
the BaseS3StorageDriver class. The practice of constructing request paths
with
parameters included needed to be removed. It causes issues during the
signature calculation. The solution is to use the params argument instead.
In addition, a fix was necessary in the signature calculation to not
compute a
payload hash in the event of file uploads. It seems this was already taken
into
consideration but failed to account for the scenario where an iterator is
passed to the data argument.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
