jfharden opened a new issue #1401: Uploading to s3 encrypted with a custom KMS key fails URL: https://github.com/apache/libcloud/issues/1401 ## Summary When uploading files to s3 where the bucket has server side encryption with a custom KMS key the upload fails ## Detailed Information Python 3.7 Libcloud: 2.8.0 OS: Amazon Linux 2 Libcloud is inspecting the etag expecting it to be the md5sum of the uploaded data object but this is not always the case (where using a custom KMS key, or doing a multi-part upload): https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html > Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an MD5 digest of their object data. > > If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption. the s3 storage driver is comparing the etag to the hash calculated locally on the streamed file (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/drivers/s3.py#L850) but where the etag is not an md5 hash of the file this will always fail I've included a stack trace below from cassandra-medusa (https://github.com/thelastpickle/cassandra-medusa) which led to me investigating this problem ``` Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 274, in main cassandra, node_backup, storage, differential_mode, config) File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 320, in do_backup num_files = backup_snapshots(storage, manifest, node_backup, node_backup_cache, snapshot) File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 388, in backup_snapshots manifest_objects = storage.storage_driver.upload_blobs(needs_backup, dst_path) File "/usr/local/lib/python3.7/site-packages/medusa/storage/s3_storage.py", line 95, in upload_blobs multi_part_upload_threshold=int(self.config.multi_part_upload_threshold), File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 87, in upload_blobs return job.execute(list(src)) File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 51, in execute return list(executor.map(self.with_storage, iterables)) File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 598, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 435, in result return self.__get_result() File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 60, in with_storage return self.func(self.storage, connection, iterable) File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 83, in <lambda> storage, connection, src_file, dest, bucket, multi_part_upload_threshold File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 119, in __upload_file obj = _upload_single_part(connection, src, bucket, full_object_name) File "/usr/local/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/local/lib/python3.7/site-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/local/lib/python3.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/local/lib/python3.7/site-packages/six.py", line 696, in reraise raise value File "/usr/local/lib/python3.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py", line 127, in _upload_single_part os.fspath(src), container=bucket, object_name=object_name File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 492, in upload_object storage_class=ex_storage_class) File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 854, in _put_object object_name=object_name, driver=self) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
