Hi,
we are currently encountering a lot of broken / orphan multipart uploads.
When I try to fetch the multipart uploads via s3cmd, it just never finishes.
Debug output looks like this and it basically never changes.
DEBUG: signature-v4 headers: {'x-amz-date': '20221202T105838Z',
'Authorization': 'XXX', 'x-amz-content-sha256': 'XXX'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(BUCKET): BUCKET.TLD
DEBUG: ConnMan.get(): re-using connection: https://BUCKET.TLD#48
DEBUG: format_uri():
/?KeyMarker=FILE&UploadIdMarker=2~nWhBa1z7eJG_7oUw-GrWZTT0CqeNdUJ&uploads
DEBUG: Sending request method_string='GET',
uri='/?KeyMarker=FILE&UploadIdMarker=2~nWhBa1z7eJG_7oUw-GrWZTT0CqeNdUJ&uploads',
headers={'x-amz-date': '20221202T105838Z', 'Authorization': 'XXX',
'x-amz-content-sha256': 'XXX'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://BUCKET.TLD#49)
DEBUG: Response:
{'data': b'<?xml version="1.0"
encoding="UTF-8"?><ListMultipartUploadsResult xm'
b'lns="http://s3.amazonaws.com/doc/2006-03-01/";><Bucket>BUCKET'
b'BUCKET</Bucket><NextKeyMarker>743ff'
b'64d-3ad6-4e3c-8816-4d8a81264657.lz4</NextKeyMarker><NextUploadIdMark'
b'er>2~nWhBa1z7eJG_7oUw-GrWZTT0CqeNdUJ</NextUploadIdMarker><MaxUploads'
b'>1000</MaxUploads><IsTruncated>true</IsTruncated><Upload><Key>FILE'
b'FILE</Key><UploadId>2~i5e5YWnSM8tGpF_a'
b'fEkr19EBzn4x09b</UploadId><Initiator><ID>USERID</ID><DisplayName>'
b'USERID</DisplayName></Initiator><Owner><ID>USERID</ID><DisplayN'
b'ame>USERID</DisplayName></Owner><StorageClass>STANDARD</StorageCla'
b'ss><Initiated>2022-11-27T08:48:36.037Z</Initiated></Upload><Upload><'
b'Key>FILE</Key><UploadId>2~nWhBa1'
b'z7eJG_7oUw-GrWZTT0CqeNdUJ</UploadId><Initiator><ID>USERID</ID><Di'
b'splayName>USERID</DisplayName></Initiator><Owner><ID>USERID</ID'
b'><DisplayName>USERID</DisplayName></Owner><StorageClass>STANDARD</'
b'StorageClass><Initiated>2022-11-28T04:38:44.781Z</Initiated></Upload'
b'></ListMultipartUploadsResult>'
Listing all files in the bucket show all files I would expect. Basically
all finished multipart uploads. In this case there are 30 files in the
bucket.
But then I check the bucket stats there 16518 and listing the bucket index
show a huge ton on _multipart_ files, that are very old.
I did a cross reference with another bucket and it only shows the actual
files in num_objects (like 7 object with a total size of 32GB)
How do I clean them up?
Last time I removed the rados objects, then I removed the omapkeys from the
index, and after that I was able to run bucket check --check-objects --fix
on that bucket.
I also tried to remove the object with radosgw-admin object rm --bucket
BUCKET --object=_multipart.... but this does not work either.
Is there a way to do it easier?
I would love to not fix 100k objects by hand.
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
