Repository: libcloud Updated Branches: refs/heads/trunk ad689715e -> 484e02a63
storage: Fix hash computation performance on upload The storage base driver computes the hash of uploaded files: individual drivers use it to ensure the reported hash is correct. Before libcloud 2.x, this was done efficiently: the file was only read once, and we were using `hash.update()` to avoid keeping the whole file in memory. With the switch to the requests module, both of these optimizations were removed inadvertently. It turns out the important one is using `hash.update()`: computing the hash on the whole file in memory is orders of magnitude slower. Closes #1135 Signed-off-by: Tomaz Muraus <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/1e955af3 Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/1e955af3 Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/1e955af3 Branch: refs/heads/trunk Commit: 1e955af39bb799d32808faa7ebe37455b8ad7e9e Parents: ad68971 Author: Quentin Pradet <[email protected]> Authored: Wed Oct 18 11:15:22 2017 +0400 Committer: Tomaz Muraus <[email protected]> Committed: Mon Jan 8 11:19:41 2018 +0100 ---------------------------------------------------------------------- libcloud/storage/base.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/libcloud/blob/1e955af3/libcloud/storage/base.py ---------------------------------------------------------------------- diff --git a/libcloud/storage/base.py b/libcloud/storage/base.py index b3de63d..617cfed 100644 --- a/libcloud/storage/base.py +++ b/libcloud/storage/base.py @@ -643,9 +643,9 @@ class StorageDriver(BaseDriver): def _hash_buffered_stream(self, stream, hasher, blocksize=65536): total_len = 0 if hasattr(stream, '__next__'): - data = libcloud.utils.files.exhaust_iterator(iterator=stream) - hasher.update(b(data)) - total_len = len(data) + for chunk in libcloud.utils.files.read_in_chunks(iterator=stream): + hasher.update(b(chunk)) + total_len += len(chunk) return (hasher.hexdigest(), total_len) if not hasattr(stream, '__exit__'): for s in stream:
