In multipart upload of S3, the checksum algorithm is different from that in
swift.
Including the following page for your convenience.
http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb
原始邮件
发件人:Saverio [email protected]
收件人:[email protected]
抄送:[email protected][email protected]
发送时间:2016年5月11日(周三) 22:15
主题:Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API atthe same
time
It does not work also the way around: If I upload a file with the swift client
with the -S options to force swift to make multipart: swift upload -S 1000000
multipart 180.mp4 Then I am not able to read the file with S3 s3cmd get
s3://multipart/180.mp4 download: 's3://multipart/180.mp4' - './180.mp4' [1 of
1] download: 's3://multipart/180.mp4' - './180.mp4' [1 of 1] 38818503 of
38818503 100% in 1s 27.32 MB/s done WARNING: MD5 signatures do not match:
computed=961f154cc78c7bf1be3b4009c29e5a68,
received=d41d8cd98f00b204e9800998ecf8427e Saverio 2016-05-11 16:07 GMT+02:00
Saverio Proto [email protected]: Thank you. It is exactly a problem with
multipart. So I tried two clients (s3cmd and rclone). When you upload a file
in S3 using multipart, you are not able to read anymore this object with the
SWIFT API because the md5 check fails. Saverio 2016-05-09 12:00 GMT+02:00
Xusangdi [email protected]: Hi, I'm not running a cluster as yours, but I
don't think the issue is caused by you using 2 APIs at the same time. IIRC the
dash thing is append by S3 multipart upload, with a following digit indicating
the number of parts. You may want to check this reported in s3cmd community:
https://sourceforge.net/p/s3tools/bugs/123/ and some basic info from Amazon:
http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html Hope this
helps :D Regards, ---Sandy -----Original Message----- From: ceph-users
[mailto:[email protected]] On Behalf Of Saverio Proto Sent:
Monday, May 09, 2016 4:42 PM To: [email protected] Subject: Re:
[ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time
I try to simplify the question to get some feedback. Is anyone running the
RadosGW in production with S3 and SWIFT API active at the same time ? thank
you ! Saverio 2016-05-06 11:39 GMT+02:00 Saverio Proto [email protected]:
Hello, We have been running the Rados GW with the S3 API and we did not
have problems for more than a year. We recently enabled also the SWIFT
API for our users. radosgw --version ceph version 0.94.6
(e832001feaf8c176593e0325c8298e3f16dfb403) The idea is that each user of
the system is free of choosing the S3 client or the SWIFT client to access
the same container/buckets. Please tell us if this is possible by design or
if we are doing something wrong. We have now a problem that some files
wrote in the past with S3, cannot be read with the SWIFT API because the
md5sum always fails. I am able to reproduce the bug in this way: We
have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know the
correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download the same
file here:
https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20
120701-ts.gz?dl=0 rclone mkdir lss3:bugreproduce rclone copy
googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce The file is
successfully uploaded. At this point I can succesfully download again the
file: rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz
test.gz but not with swift: swift download googlebooks-ngrams-gz
fre/googlebooks-fre-all-2gram-20120701-ts.gz Error downloading object
'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz': u'Error
downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz: md5sum != etag,
1c8113d2bd21232688221ec74dccff3a != 1a209a31b4ac3eb923fac5e8d194d9d3-2'
Also I found strange the dash character '-' at the end of the md5 that is
trying to compare. Of course upload a file with the swift client and
redownloading the same file just works. Should I open a bug for the
radosgw on http://tracker.ceph.com/ ? thank you Saverio
_______________________________________________ ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ This e-mail and its attachments
contain confidential information from H3C, which is intended only for the
person or entity whose address is listed above. Any use of the information
contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify
the sender by phone or email immediately and delete it!
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
