c-w opened a new pull request #1400: Implemented chunked upload for Azure Blobs storage driver URL: https://github.com/apache/libcloud/pull/1400 ## Implemented chunked upload for Azure Blobs storage driver ### Description This pull request fixes https://github.com/apache/libcloud/issues/1399 by implementing chunked upload in the Azure Blobs storage driver via the [Put Block](https://docs.microsoft.com/en-us/rest/api/storageservices/put-block) and [Put Block List](https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list) APIs. The implementation mostly leverages the `AzureBlobsStorageDriver._upload_in_chunks` function that was first introduced in [24f34c9](https://github.com/apache/libcloud/blob/24f34c99c9440523a53e940a346bced551281953/libcloud/storage/drivers/azure_blobs.py#L732-L788) and was stopped being used in [6e0040d](https://github.com/apache/libcloud/commit/6e0040d8904cacb5dbe88309e9051be08cdc59f9) with the the following main updates: - Enable setting object metadata, content-type and content-md5 for chunked uploads. - Drop support for PageBlob. **This is a breaking API change.** However, given the non-trivial differences between the PageBlob and BlockBlob APIs, I'd hypothesize that the improved simplicity of the code will aid in the long run with bugfixes and maintenance. If keeping support for PageBlob is a hard requirement, I suspect a non-trivial amount of refactoring would have to be undertaken to cleanly support chunked upload for both BlockBlob and PageBlob object types. I'm open to do the work as part of this pull request, but I'd love to first discuss the pros/cons of both approaches. Additionally, the following companion changes were made: - Remove unused `upload_func` and `upload_func_kwargs` arguments in `StorageDriver._upload_object` ([7e55057](https://github.com/apache/libcloud/commit/7e550577758cb95ec42819e77b6773566a180e5c)) - Rationalize determination of content type to a new helper function `StorageDriver._determine_content_type` ([0dc5cbb](https://github.com/apache/libcloud/commit/0dc5cbb0410fa3b1f9ca0d54b59d514cbc7e6c7a)) - Switch `libcloud.utils.files.read_in_chunks` from type checking to duck-typing to ensure that more iterators (e.g. opened file objects) use the fast `read(int)` code-path ([0fded11](https://github.com/apache/libcloud/commit/0fded110e69f020388d76d8094d3ca25fa868e13)) ### Status - work in progress ### Checklist - [x] Code linting ([CI passed](https://travis-ci.org/CatalystCode/libcloud/builds/632547395)) - [ ] Documentation (*TODO: changelog entry*) - [ ] Tests ([E2E tests passed](https://clewolff.visualstudio.com/libcloud-tests/_build/results?buildId=661), *TODO: unit tests*) - [x] ICLA ([committer](http://people.apache.org/phonebook.html?uid=clewolff))
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
