Public bug reported:
We are trying to upload images to an on-premise minio-based S3
compatible storage cluster. We have been using it for ~30GB images for
months. However when increasing the size to >30GB, the image is
successfully uploaded and then glance seems to tell it to delete.
Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812 1 1 rPR1E4MBBdpF5mC-rsYE
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 6, 2022 @ 08:39:39.038 1 1 q_R1E4MBBdpF5mC-A8YF
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
...
Whole bunch of PutObjectPart...
Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067 1 1 PvTnAIMBBdpF5mC-XLv7
test_logs - ttcloud-image ttcloud-image DeleteObject
DeleteObject
Sep 2, 2022 @ 18:11:45.987 1 1 PfTnAIMBBdpF5mC-XLur
test_logs - ttcloud-image ttcloud-image HeadObject
HeadObject
Sep 2, 2022 @ 18:11:45.927 1 1 PPTnAIMBBdpF5mC-XLtw
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 2, 2022 @ 18:10:49.370 1 1 O_TmAIMBBdpF5mC-f7uE
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
We were able to get the 38GB image to work by increasing the chunk size from
10MB to 100MB. However this does not help the 50GB or 100GB.
The storage system is correctly responding to all S3 request with either
200 or 204 replies. It appears that something important is happening
with the HeadObject which is then causing glance to send a DeleteObject.
I've attached shortened versions of the S3 logs - I've deleted the
majority of PutObjectPart entries to keep the size down.
We're not finding any information. What is causing this behavior?
** Affects: glance
Importance: Undecided
Status: New
** Tags: complete delete deleteobject headobject s3
** Attachment added: "S3 Logs"
https://bugs.launchpad.net/bugs/1989013/+attachment/5614450/+files/S3%20-%20Fail%20+38GB%20S3%20Transfer%20%28short%29.numbers
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1989013
Title:
Large images uploaded but then deleted
Status in Glance:
New
Bug description:
We are trying to upload images to an on-premise minio-based S3
compatible storage cluster. We have been using it for ~30GB images for
months. However when increasing the size to >30GB, the image is
successfully uploaded and then glance seems to tell it to delete.
Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812 1 1 rPR1E4MBBdpF5mC-rsYE
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 6, 2022 @ 08:39:39.038 1 1 q_R1E4MBBdpF5mC-A8YF
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
...
Whole bunch of PutObjectPart...
Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067 1 1 PvTnAIMBBdpF5mC-XLv7
test_logs - ttcloud-image ttcloud-image DeleteObject
DeleteObject
Sep 2, 2022 @ 18:11:45.987 1 1 PfTnAIMBBdpF5mC-XLur
test_logs - ttcloud-image ttcloud-image HeadObject
HeadObject
Sep 2, 2022 @ 18:11:45.927 1 1 PPTnAIMBBdpF5mC-XLtw
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 2, 2022 @ 18:10:49.370 1 1 O_TmAIMBBdpF5mC-f7uE
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
We were able to get the 38GB image to work by increasing the chunk size from
10MB to 100MB. However this does not help the 50GB or 100GB.
The storage system is correctly responding to all S3 request with
either 200 or 204 replies. It appears that something important is
happening with the HeadObject which is then causing glance to send a
DeleteObject.
I've attached shortened versions of the S3 logs - I've deleted the
majority of PutObjectPart entries to keep the size down.
We're not finding any information. What is causing this behavior?
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1989013/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
