I suspect this is a glance_store issue, but keeping both components
until more research is done.
** Also affects: glance-store
Importance: Undecided
Status: New
** Summary changed:
- Large images uploaded but then deleted
+ s3: Large images uploaded but then deleted
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1989013
Title:
s3: Large images uploaded but then deleted
Status in Glance:
New
Status in glance_store:
New
Bug description:
We are trying to upload images to an on-premise minio-based S3
compatible storage cluster. We have been using it for ~30GB images for
months. However when increasing the size to >30GB, the image is
successfully uploaded and then glance seems to tell it to delete.
Successful log looks like this:
Sep 6, 2022 @ 08:40:22.812 1 1 rPR1E4MBBdpF5mC-rsYE
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 6, 2022 @ 08:39:39.038 1 1 q_R1E4MBBdpF5mC-A8YF
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
...
Whole bunch of PutObjectPart...
Failure log looks like this:
Sep 2, 2022 @ 18:11:46.067 1 1 PvTnAIMBBdpF5mC-XLv7
test_logs - ttcloud-image ttcloud-image DeleteObject
DeleteObject
Sep 2, 2022 @ 18:11:45.987 1 1 PfTnAIMBBdpF5mC-XLur
test_logs - ttcloud-image ttcloud-image HeadObject
HeadObject
Sep 2, 2022 @ 18:11:45.927 1 1 PPTnAIMBBdpF5mC-XLtw
test_logs - ttcloud-image ttcloud-image CompleteMultipartUpload
CompleteMultipartUpload
Sep 2, 2022 @ 18:10:49.370 1 1 O_TmAIMBBdpF5mC-f7uE
test_logs - ttcloud-image ttcloud-image PutObjectPart
PutObjectPart
We were able to get the 38GB image to work by increasing the chunk size from
10MB to 100MB. However this does not help the 50GB or 100GB.
The storage system is correctly responding to all S3 request with
either 200 or 204 replies. It appears that something important is
happening with the HeadObject which is then causing glance to send a
DeleteObject.
I've attached shortened versions of the S3 logs - I've deleted the
majority of PutObjectPart entries to keep the size down.
We're not finding any information. What is causing this behavior?
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1989013/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
